Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
432.788584ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:42:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 432.814698ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................................X..X............] duration:
518.15029ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:42:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 518.173486ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X.X............] duration:
340.838793ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:42:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 340.863613ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................XX..X.X.............] duration:
463.510372ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:42:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 463.611966ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X.....X............] duration:
421.983413ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:42:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 422.012992ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X.....X...........] duration:
352.177242ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:42:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 352.278093ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...........................................X...XX.X.........] duration:
390.271181ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:41:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 390.298543ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
428.446504ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:41:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 428.474763ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................................XX..X.X............] duration:
432.037063ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:41:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 432.116181ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration:
363.448213ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:41:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 363.473438ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X.....X..........] duration:
427.214726ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:41:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 427.243784ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X..X...........] duration:
528.209595ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:41:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 528.231335ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration:
471.949126ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:41:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 471.974062ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X.....X.........] duration:
350.181804ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:41:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 350.205587ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
376.508596ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:41:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 376.539837ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............................................X...XX.X........] duration:
410.609399ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:41:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 410.633216ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................................XX..X.X...........] duration:
452.481357ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:41:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 452.511329ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X..X..........] duration:
505.374697ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:41:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 505.955301ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................................XX..X.X..........] duration:
464.77692ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:40:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 464.802961ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............................................X...XX.X.......] duration:
438.107299ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:40:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 438.13018ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X.X.........] duration:
386.992382ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:40:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 387.018209ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X.....X........] duration:
457.703408ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:40:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 457.725704ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................................XX..X.X.........] duration:
531.025366ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:40:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 531.129013ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
359.753708ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:39:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 359.786804ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X..X.........] duration:
692.016607ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:39:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 692.073457ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X.X........] duration:
377.01995ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:39:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 377.044442ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................................X..X........] duration:
526.063733ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:39:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 526.084187ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............................................X...XX.X......] duration:
397.873674ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:39:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 397.914824ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration:
378.924445ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:39:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 378.951416ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................................XX..X.X........] duration:
532.716773ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:39:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 532.747042ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
453.796246ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:39:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 453.822644ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X..X.......] duration:
543.543234ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:39:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 543.569898ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X.....X.......] duration:
434.308498ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:38:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 434.386041ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................................XX..X.X.......] duration:
440.221034ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:38:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 440.254856ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............................................X...XX.X.....] duration:
587.50178ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:38:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 587.522292ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
332.433581ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:38:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 332.48244ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X.X......] duration:
388.313336ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:38:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 388.334537ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X..X......] duration:
544.212544ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:38:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 544.246095ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X.....X......] duration:
427.341722ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:37:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 427.393519ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................................XX..X.X......] duration:
501.825717ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:37:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.868474ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [................................................X...XX.X....] duration:
411.84936ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:37:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 411.876942ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration:
369.008821ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:37:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 369.03708ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X..X.....] duration:
509.406229ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:37:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 509.440662ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
370.12678ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:37:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 370.149588ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X..X....] duration:
520.182858ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:36:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 520.209785ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
332.369655ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:36:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 332.39277ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................................XX..X.X.....] duration:
453.613323ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:36:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 453.638605ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.................................................X...XX.X...] duration:
417.393597ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:36:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 417.427519ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X.....X.....] duration:
349.55256ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:36:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 349.595698ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X.....X....] duration:
355.766174ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:36:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 355.79183ms
core
uplink
us-select-1
us1-placement-20
|