Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................XX..X............................] duration: 467.015973ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:29:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 467.093691ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 419.209275ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:29:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 419.291662ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................XX..X...........................] duration: 433.367341ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 433.392455ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 452.743598ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 452.775028ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................XX..X..........................] duration: 488.160363ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:28:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.155902ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 477.909355ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:27:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 477.937005ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 500.365574ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:27:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 500.389879ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 452.62656ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:27:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 452.654342ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................XX..X.........................] duration: 448.585227ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:27:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.685136ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 388.562097ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:26:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 388.587599ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 402.154085ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:26:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 402.250023ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................XX..X........................] duration: 393.417ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:26:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.44451ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 457.595917ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:26:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 457.620316ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 411.347963ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:25:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 411.372036ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 542.724151ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:25:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 542.7471ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................XX..X.......................] duration: 470.661279ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:25:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.747967ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 475.811044ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:25:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 475.834854ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................XX..X......................] duration: 445.986897ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:25:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.010646ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 398.496101ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:24:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 398.522999ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 492.995639ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:24:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 493.020613ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 401.41224ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:24:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 401.442583ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................XX..X.....................] duration: 429.019992ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:24:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 429.042331ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 426.705447ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:24:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 426.759324ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 439.170483ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:23:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 439.200472ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 399.407155ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:23:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 399.432902ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................XX..X....................] duration: 431.00359ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:23:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 431.034413ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 434.540227ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:22:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 434.577584ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................XX..X...................] duration: 442.529224ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:22:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 442.575527ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 482.147931ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:22:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 482.17987ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 383.344145ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:22:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 383.371175ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................XX..X..................] duration: 461.410705ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:21:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.468587ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 494.475072ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 494.504477ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 504.293495ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:21:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 504.321865ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 458.006222ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:20:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 458.042146ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................XX..X.................] duration: 459.231796ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:20:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 459.283291ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 406.998598ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:20:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 407.023432ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................................XX..X................] duration: 457.165434ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:19:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.194319ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 509.847579ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:19:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 509.870429ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 443.708622ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:19:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 443.738421ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.020307165s trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:19:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.020386899s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................XX..X...............] duration: 431.180811ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:19:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 431.204645ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 474.14104ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:18:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 474.172306ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 412.413023ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:18:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 412.44867ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................XX..X..............] duration: 490.099267ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:18:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 490.161853ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 520.775874ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:18:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 520.801363ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 455.412507ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:17:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 455.438199ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 512.283294ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:17:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 512.310934ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 460.608062ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:17:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 460.627797ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................................XX..X.............] duration: 466.27086ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:17:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 466.316881ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 582.095384ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 582.135614ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)