Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 460.608062ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:17:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 460.627797ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................................XX..X.............] duration: 466.27086ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:17:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 466.316881ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 582.095384ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 582.135614ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................................XX..X............] duration: 457.352621ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:16:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.377416ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 433.175804ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:16:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 433.205899ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 501.36247ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:16:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 501.386577ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 409.183002ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:16:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 409.211359ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................................XX..X...........] duration: 456.151875ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:15:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.16904ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 498.567036ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:15:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 498.663527ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 414.990737ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:15:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 415.053006ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 494.696241ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:14:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 494.794313ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................................XX..X..........] duration: 446.613157ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:14:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.640054ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 419.61811ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:14:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 419.641531ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 439.635416ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:14:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 439.660651ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 511.067369ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:13:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 511.098123ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................................XX..X.........] duration: 468.521781ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:13:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 468.551645ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 363.310768ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:13:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 363.340344ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 556.609921ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:12:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 556.632848ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 433.83478ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:12:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 433.84805ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................................XX..X........] duration: 526.496178ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:12:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 526.520401ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 410.482089ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:12:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 410.607758ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 521.631049ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:12:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 521.716224ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................................XX..X.......] duration: 444.833056ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:12:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.866192ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................................XX..X......] duration: 439.886459ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:11:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 439.907167ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................................XX..X.....] duration: 434.250752ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:11:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.271266ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 413.430749ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:11:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 413.458973ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 465.213332ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:11:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 465.229877ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 503.991511ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 504.021377ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................................XX..X....] duration: 432.257108ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:10:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 432.274268ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 505.888907ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:10:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 505.91469ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................................XX..X...] duration: 457.694731ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:10:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.743522ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 496.255597ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:10:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 496.289866ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 371.705887ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:09:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 371.730614ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................................XX..X..] duration: 442.855406ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:09:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 442.907789ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 480.193747ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:09:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 480.225756ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 379.462476ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:08:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 379.507361ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................................XX..X.] duration: 442.799511ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:08:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 442.830862ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 464.762277ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:08:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 464.779416ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 458.867148ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:08:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 458.895296ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 31.872358271s trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.872384921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 34.176684827s trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.176710664s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................................................XX..X] duration: 5.000597421s trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:07:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00062075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 532.925854ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:07:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 532.950617ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................XX..] duration: 435.762754ms trace: https://jaeger.dev.storj.io/trace/7d49b13549bf3a30 FAIL (2025-12-03 03:07:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 435.787031ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 379.058005ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:06:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 379.081916ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 569.073325ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:06:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 569.09996ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................XX.] duration: 533.118814ms trace: https://jaeger.dev.storj.io/trace/7d49b13549bf3a30 FAIL (2025-12-03 03:06:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.168469ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 456.545737ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:06:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 456.569586ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 653.871182ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:06:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 653.892837ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.016894114s trace: https://jaeger.dev.storj.io/trace/7d49b13549bf3a30 FAIL (2025-12-03 03:06:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.016923941s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)