Orbiter History

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 35.792703ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:19:51 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 35.817469ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.................................................X.XXX.XX...] duration: 2.940243586s FAIL (2025-08-07 23:19:49 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.940273538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XX.X.X...] duration: 833.774801ms FAIL (2025-08-07 23:19:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 833.798041ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..................................................X..XXXXX.X] duration: 30.000611054s FAIL (2025-08-07 23:19:40 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000637409s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXXXX.....] duration: 1.79338667s FAIL (2025-08-07 23:19:34 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.793434933s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 258.925513ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:19:34 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 258.94728ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 405.649072ms FAIL (2025-08-07 23:19:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:11:04.814410393 (13h31m39.329512969s) [upload=13925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:11:04.814415016 (13h31m39.329517592s) [upload=13925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:04.947583472 (13h31m39.462686071s) [upload=13925]: Done uploading segments... Aug 7 23:11:04.947609736 (13h31m39.462712324s) [upload=13925]: Upload finished. err: <nil> Aug 7 23:11:29.227536921 (13h32m3.742639487s) [upload=13931]: Starting upload Aug 7 23:11:29.227600047 (13h32m3.742702623s) [upload=13931]: Done starting upload Aug 7 23:11:29.227715021 (13h32m3.742817591s) [upload=13931]: Uploading segments... Aug 7 23:11:29.227850209 (13h32m3.74295278s) [upload=13931]: Got next segment. Inline: false Aug 7 23:11:29.227898592 (13h32m3.743001169s) [upload=13931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:29.227903365 (13h32m3.743005942s) [upload=13931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:11:47.514990540 (13h32m22.03009312s) [seg_pos={0 0}, upload=13931]: Begin upload segment... Aug 7 23:11:47.515067094 (13h32m22.030169657s) [seg_pos={0 0}, upload=13931]: Done begin upload segment. Aug 7 23:11:47.515129228 (13h32m22.030231814s) [noise=true , address=77.90.31.18:4806 , piece_id=LI5HUM3HCYFYYSO6BENU4DQ2SFOPEJLRWZRM5PWCFXUZON4WCMAA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.515554241 (13h32m22.03065682s) [noise=true , address=104.36.82.14:4222 , piece_id=B5WB353FQ47THJP7MXV4AY5Z6FFSG7A66X72BLJUDK2ZTM3FNDVA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.516068554 (13h32m22.031171143s) [noise=true , address=45.91.116.7:4010 , piece_id=XOROULN3EEUSG45LZ5V53ORP2YV7CWLRQDSQ2AXREAFB62FGNL6A, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.516625763 (13h32m22.031728346s) [noise=true , address=45.91.116.8:4047 , piece_id=5E2VVSY65OAD6OFCQBBOTYLCLUKNVCVD3KY7OOVBI5TCYKQUKBQQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517060179 (13h32m22.03216276s) [noise=true , address=104.36.82.7:4910 , piece_id=Y75KUQR2IYRZBANBQPDFZTUJZPM54BA3KI6ZEF77NI3V755HPBOA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517396640 (13h32m22.032499217s) [noise=true , address=104.36.82.6:4873 , piece_id=3YGEWHQ7MXU6IPUJCYNM7L4DMNOECDSTVXMUBCX3CJCGPJ4OGGIQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517801490 (13h32m22.032904072s) [noise=true , address=45.91.116.10:4081 , piece_id=WDJ6KCU4LP75N7TSSR2VTWWEON3VRWE7BFJL57H2RM7ZYI2GYLWQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518135215 (13h32m22.033237797s) [noise=true , address=45.91.116.5:4297 , piece_id=T76WY44JQ55TJH4KJM57QWNBP7DVYNIBDVAPVN2AUTWASN4V76IA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518441967 (13h32m22.033544543s) [noise=true , address=77.90.31.11:4553 , piece_id=NK6AJDEDIAA5N3DNKBIPKZXURRSBOJG26FIMUB2SVFTIWOKFJWNQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518756911 (13h32m22.033859486s) [noise=true , address=77.90.31.21:4877 , piece_id=NVD7UBISRZJFOBJR7JKOX6V354Z6VCEP5TWJPGW3WOSI6S6K5FDA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519217404 (13h32m22.034319986s) [noise=true , address=45.91.116.6:4973 , piece_id=HEA5GQ53KZVIMBZ5LC3S4Y4372WMGUSATCBDTY5BIHZ4WL2PG7HQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519516685 (13h32m22.034619271s) [noise=true , address=77.90.31.19:4841 , piece_id=Z76CMQDB3DDLHQV35DZORZWRBX2D52BLBDCKPFXPONE3OUNGSGHA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519904148 (13h32m22.035006727s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7PUVWG4BXEJSS4YOBVTMUHCAWWKKQAR43NH4U3R24YMK3MP6H6Q, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.520220651 (13h32m22.035323239s) [noise=true , address=77.90.31.10:4517 , piece_id=WJ7TMWHMURKZ6HJON2NBIZVIC4XEXUUR5M7KD5PFWV4MEB3DUV7A, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.520644038 (13h32m22.035746619s) [noise=true , address=104.36.82.12:4149 , piece_id=UEFYG73QPD3OQX5IF2BGE7SQHYL5OKV37LQOKGNECBGRYQLULCJQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.521219880 (13h32m22.036322466s) [noise=true , address=77.90.31.17:4771 , piece_id=YCZ26U3QI4NQ7CKNVZEEB4O6XK4ZR3SPXTKT75P5U4ZQLUWOJWQA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.609483246 (13h32m22.124585826s) [upload=13931]: Next returned nil segment Aug 7 23:11:47.609501768 (13h32m22.124604348s) [upload=13931]: Waiting for error group managing segments... Aug 7 23:11:47.614240475 (13h32m22.129343046s) [noise=true , address=45.91.116.8:4047 , piece_id=5E2VVSY65OAD6OFCQBBOTYLCLUKNVCVD3KY7OOVBI5TCYKQUKBQQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.614925369 (13h32m22.130027939s) [noise=true , address=45.91.116.5:4297 , piece_id=T76WY44JQ55TJH4KJM57QWNBP7DVYNIBDVAPVN2AUTWASN4V76IA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.615059457 (13h32m22.130162026s) [noise=true , address=45.91.116.7:4010 , piece_id=XOROULN3EEUSG45LZ5V53ORP2YV7CWLRQDSQ2AXREAFB62FGNL6A, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.615458775 (13h32m22.130561338s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7PUVWG4BXEJSS4YOBVTMUHCAWWKKQAR43NH4U3R24YMK3MP6H6Q, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.616116276 (13h32m22.131218838s) [noise=true , address=77.90.31.18:4806 , piece_id=LI5HUM3HCYFYYSO6BENU4DQ2SFOPEJLRWZRM5PWCFXUZON4WCMAA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.616661348 (13h32m22.131763909s) [noise=true , address=77.90.31.17:4771 , piece_id=YCZ26U3QI4NQ7CKNVZEEB4O6XK4ZR3SPXTKT75P5U4ZQLUWOJWQA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.618000416 (13h32m22.133102978s) [noise=true , address=45.91.116.10:4081 , piece_id=WDJ6KCU4LP75N7TSSR2VTWWEON3VRWE7BFJL57H2RM7ZYI2GYLWQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.618530300 (13h32m22.133632866s) [noise=true , address=45.91.116.6:4973 , piece_id=HEA5GQ53KZVIMBZ5LC3S4Y4372WMGUSATCBDTY5BIHZ4WL2PG7HQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.619098772 (13h32m22.13420134s) [noise=true , address=77.90.31.19:4841 , piece_id=Z76CMQDB3DDLHQV35DZORZWRBX2D52BLBDCKPFXPONE3OUNGSGHA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.619697172 (13h32m22.13479975s) [noise=true , address=77.90.31.11:4553 , piece_id=NK6AJDEDIAA5N3DNKBIPKZXURRSBOJG26FIMUB2SVFTIWOKFJWNQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.620954558 (13h32m22.136057128s) [noise=true , address=77.90.31.21:4877 , piece_id=NVD7UBISRZJFOBJR7JKOX6V354Z6VCEP5TWJPGW3WOSI6S6K5FDA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.627010966 (13h32m22.142113543s) [noise=true , address=104.36.82.7:4910 , piece_id=Y75KUQR2IYRZBANBQPDFZTUJZPM54BA3KI6ZEF77NI3V755HPBOA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.629783113 (13h32m22.144885677s) [noise=true , address=77.90.31.10:4517 , piece_id=WJ7TMWHMURKZ6HJON2NBIZVIC4XEXUUR5M7KD5PFWV4MEB3DUV7A, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.629816031 (13h32m22.144918609s) [seg_pos={0 0}, upload=13931]: Segment reached optimal threshold of 13 pieces. Aug 7 23:11:47.630126623 (13h32m22.145229201s) [noise=true , address=104.36.82.14:4222 , piece_id=B5WB353FQ47THJP7MXV4AY5Z6FFSG7A66X72BLJUDK2ZTM3FNDVA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection Aug 7 23:11:47.630137290 (13h32m22.145239874s) [noise=true , address=104.36.82.12:4149 , piece_id=UEFYG73QPD3OQX5IF2BGE7SQHYL5OKV37LQOKGNECBGRYQLULCJQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection; write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection Aug 7 23:11:47.630232787 (13h32m22.145335361s) [noise=true , address=104.36.82.6:4873 , piece_id=3YGEWHQ7MXU6IPUJCYNM7L4DMNOECDSTVXMUBCX3CJCGPJ4OGGIQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 159.223.143.39:36062->104.36.82.6:4873: use of closed network connection; context canceled Aug 7 23:11:47.630264689 (13h32m22.145367273s) [seg_pos={0 0}, upload=13931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:11:47.630345104 (13h32m22.145447682s) [upload=13931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:11:47.630350066 (13h32m22.145452652s) [upload=13931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:37.334152859 (13h39m11.849255428s) [upload=13931]: Done uploading segments... Aug 7 23:18:37.334173589 (13h39m11.849276166s) [upload=13931]: Upload finished. err: metaclient: context deadline exceeded Duration: 405.677744ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XX..X...] duration: 259.262576ms FAIL (2025-08-07 23:19:16 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad finished. err: <nil> Aug 7 23:07:56.297738414 (13h28m30.812840994s) [upload=13869]: Starting upload Aug 7 23:07:56.297858483 (13h28m30.812961062s) [upload=13869]: Done starting upload Aug 7 23:07:56.297993444 (13h28m30.813096021s) [upload=13869]: Uploading segments... Aug 7 23:07:56.298078193 (13h28m30.813180778s) [upload=13869]: Got next segment. Inline: true Aug 7 23:07:56.298102093 (13h28m30.813204678s) [upload=13869]: Waiting for error group managing segments... Aug 7 23:07:56.298156325 (13h28m30.813258907s) [upload=13869]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:56.298177912 (13h28m30.813280494s) [upload=13869]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:56.298182546 (13h28m30.813285127s) [upload=13869]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:56.372043207 (13h28m30.887145784s) [upload=13869]: Done uploading segments... Aug 7 23:07:56.372061460 (13h28m30.887164036s) [upload=13869]: Upload finished. err: <nil> Aug 7 23:08:37.049508399 (13h29m11.564610963s) [upload=13879]: Starting upload Aug 7 23:08:37.049588017 (13h29m11.564690596s) [upload=13879]: Done starting upload Aug 7 23:08:37.049683838 (13h29m11.564786416s) [upload=13879]: Uploading segments... Aug 7 23:08:37.049829834 (13h29m11.564932424s) [upload=13879]: Got next segment. Inline: true Aug 7 23:08:37.049839426 (13h29m11.564942003s) [upload=13879]: Waiting for error group managing segments... Aug 7 23:08:37.049910072 (13h29m11.565012644s) [upload=13879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:37.049916484 (13h29m11.56501906s) [upload=13879]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:37.049920022 (13h29m11.565022598s) [upload=13879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:37.140370137 (13h29m11.655472717s) [upload=13879]: Done uploading segments... Aug 7 23:08:37.141250272 (13h29m11.65635286s) [upload=13879]: Upload finished. err: <nil> Aug 7 23:09:29.216369700 (13h30m3.731472269s) [upload=13895]: Starting upload Aug 7 23:09:29.216489949 (13h30m3.731592525s) [upload=13895]: Done starting upload Aug 7 23:09:29.216567331 (13h30m3.731669896s) [upload=13895]: Uploading segments... Aug 7 23:09:29.216612331 (13h30m3.731714917s) [upload=13895]: Got next segment. Inline: true Aug 7 23:09:29.216618125 (13h30m3.731720692s) [upload=13895]: Waiting for error group managing segments... Aug 7 23:09:29.216659267 (13h30m3.731761843s) [upload=13895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:29.216663938 (13h30m3.731766517s) [upload=13895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:29.216668553 (13h30m3.731771139s) [upload=13895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:29.297928170 (13h30m3.813030739s) [upload=13895]: Done uploading segments... Aug 7 23:09:29.297957487 (13h30m3.813060063s) [upload=13895]: Upload finished. err: <nil> Aug 7 23:10:23.115383547 (13h30m57.630486132s) [upload=13915]: Starting upload Aug 7 23:10:23.115456203 (13h30m57.630558786s) [upload=13915]: Done starting upload Aug 7 23:10:23.115557871 (13h30m57.630660452s) [upload=13915]: Uploading segments... Aug 7 23:10:23.115614526 (13h30m57.630717116s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:23.115621076 (13h30m57.630723665s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:23.115685040 (13h30m57.630787627s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:23.115692213 (13h30m57.630794802s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:23.115697119 (13h30m57.630799709s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:23.183146377 (13h30m57.698248962s) [upload=13915]: Done uploading segments... Aug 7 23:10:23.183169407 (13h30m57.698271988s) [upload=13915]: Upload finished. err: <nil> Aug 7 23:10:46.162725354 (13h31m20.67782793s) [upload=13919]: Starting upload Aug 7 23:10:46.162807728 (13h31m20.677910304s) [upload=13919]: Done starting upload Aug 7 23:10:46.162877771 (13h31m20.677980347s) [upload=13919]: Uploading segments... Aug 7 23:10:46.162939319 (13h31m20.678041901s) [upload=13919]: Got next segment. Inline: true Aug 7 23:10:46.162944044 (13h31m20.678046624s) [upload=13919]: Waiting for error group managing segments... Aug 7 23:10:46.162986704 (13h31m20.678089286s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:46.162991049 (13h31m20.678093629s) [upload=13919]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:46.162994213 (13h31m20.678096792s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:49.331481110 (13h31m23.84658369s) [upload=13919]: Done uploading segments... Aug 7 23:10:49.331502710 (13h31m23.846605285s) [upload=13919]: Upload finished. err: <nil> Aug 7 23:11:56.588799333 (13h32m31.103901909s) [upload=13942]: Starting upload Aug 7 23:11:56.588871172 (13h32m31.103973747s) [upload=13942]: Done starting upload Aug 7 23:11:56.588959193 (13h32m31.104061755s) [upload=13942]: Uploading segments... Aug 7 23:11:56.588996486 (13h32m31.104099072s) [upload=13942]: Got next segment. Inline: true Aug 7 23:11:56.589000914 (13h32m31.104103491s) [upload=13942]: Waiting for error group managing segments... Aug 7 23:11:56.589047870 (13h32m31.104150433s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.589051856 (13h32m31.104154432s) [upload=13942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.589054372 (13h32m31.10415695s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:56.555383755 (13h33m31.070486344s) [upload=13942]: Done uploading segments... Aug 7 23:12:56.555407795 (13h33m31.070510378s) [upload=13942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:56.669023639 (13h33m31.184126201s) [upload=13959]: Starting upload Aug 7 23:12:56.669081512 (13h33m31.184184088s) [upload=13959]: Done starting upload Aug 7 23:12:56.669139265 (13h33m31.184241827s) [upload=13959]: Uploading segments... Aug 7 23:12:56.669181725 (13h33m31.184284307s) [upload=13959]: Got next segment. Inline: true Aug 7 23:12:56.669185727 (13h33m31.18428829s) [upload=13959]: Waiting for error group managing segments... Aug 7 23:12:56.669222629 (13h33m31.184325191s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:56.669226085 (13h33m31.18432866s) [upload=13959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:56.669228556 (13h33m31.184331118s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.639927224 (13h34m31.155029791s) [upload=13959]: Done uploading segments... Aug 7 23:13:56.639946863 (13h34m31.155049429s) [upload=13959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:56.754611797 (13h34m31.269714367s) [upload=13974]: Starting upload Aug 7 23:13:56.754690544 (13h34m31.269793123s) [upload=13974]: Done starting upload Aug 7 23:13:56.754787910 (13h34m31.269890478s) [upload=13974]: Uploading segments... Aug 7 23:13:56.754829279 (13h34m31.269931864s) [upload=13974]: Got next segment. Inline: true Aug 7 23:13:56.754833834 (13h34m31.269936411s) [upload=13974]: Waiting for error group managing segments... Aug 7 23:13:56.754887650 (13h34m31.269990217s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.754891993 (13h34m31.26999456s) [upload=13974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.754895495 (13h34m31.269998071s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.814228870 (13h34m31.32933145s) [upload=13974]: Done uploading segments... Aug 7 23:13:56.814253732 (13h34m31.329356295s) [upload=13974]: Upload finished. err: <nil> Aug 7 23:14:51.237083244 (13h35m25.75218581s) [upload=13984]: Starting upload Aug 7 23:14:51.237154440 (13h35m25.752257016s) [upload=13984]: Done starting upload Aug 7 23:14:51.237222602 (13h35m25.752325168s) [upload=13984]: Uploading segments... Aug 7 23:14:51.237272573 (13h35m25.752375158s) [upload=13984]: Got next segment. Inline: true Aug 7 23:14:51.237277088 (13h35m25.752379654s) [upload=13984]: Waiting for error group managing segments... Aug 7 23:14:51.237328372 (13h35m25.752430937s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:51.237332974 (13h35m25.75243555s) [upload=13984]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:51.237336117 (13h35m25.752438684s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:51.367347705 (13h35m25.882450281s) [upload=13984]: Done uploading segments... Aug 7 23:14:51.367368302 (13h35m25.882470865s) [upload=13984]: Upload finished. err: <nil> Aug 7 23:16:05.237199068 (13h36m39.752301655s) [upload=14002]: Starting upload Aug 7 23:16:05.237269888 (13h36m39.752372457s) [upload=14002]: Done starting upload Aug 7 23:16:05.237360382 (13h36m39.752462952s) [upload=14002]: Uploading segments... Aug 7 23:16:05.237841750 (13h36m39.752944336s) [upload=14002]: Got next segment. Inline: true Aug 7 23:16:05.237859632 (13h36m39.752962218s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:16:05.237930004 (13h36m39.753032586s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.237936533 (13h36m39.753039109s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.237940573 (13h36m39.753043148s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.205359206 (13h37m39.720461782s) [upload=14002]: Done uploading segments... Aug 7 23:17:05.205376470 (13h37m39.720479046s) [upload=14002]: Upload finished. err: metaclient: context deadline exceeded Duration: 259.297481ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.576373061s remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:19:16 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 3.576400785s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XX.XXXX...] duration: 484.807718ms FAIL (2025-08-07 23:19:14 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 783045 (13h33m55.532885624s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.018198306 (13h33m55.533300869s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.099506775 (13h33m55.614609341s) [upload=13963]: Next returned nil segment Aug 7 23:13:21.099521270 (13h33m55.614623846s) [upload=13963]: Waiting for error group managing segments... Aug 7 23:13:21.104019028 (13h33m55.619121597s) [noise=true , address=45.91.116.5:4305 , piece_id=V2T7C5TPEPR4KDVZI67JO2DIIQGXNTCLOBEOOIZYGQYN4TB5UCKQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.109200706 (13h33m55.624303283s) [noise=true , address=104.36.82.17:4352 , piece_id=2FNMB25WSQMNVTAP7UCNXFJQJLF7VXVWSVOQLN2CTF4N6GQUQAGA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.110092715 (13h33m55.625195286s) [noise=true , address=104.36.82.43:4837 , piece_id=TCE3IJZU7ZA4Z5ONPBDQANINXZXA3BHKORRZGBB7AN6WZULACQ5A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.111990406 (13h33m55.627092973s) [noise=true , address=104.36.82.62:4435 , piece_id=3XPVD4VZ43PTPELTHNJHSMYCD4CIZGG3DBDBXDG247J22HYT6O2Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.112988613 (13h33m55.628091177s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.113711920 (13h33m55.628814482s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114233050 (13h33m55.629335628s) [noise=true , address=104.36.82.38:4712 , piece_id=6G4TGYY3JILWXGGVFMLX3BO77QRHM4OXVXJNL3YMAD53EU55MZXA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114540807 (13h33m55.629643369s) [noise=true , address=45.91.116.8:4053 , piece_id=AB4MH3EBI2HTBZH6UZXJXHWPP7SK6TF6QAE6NHGVIJ363VOY3DVQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.122184649 (13h33m55.637287217s) [noise=true , address=104.36.82.44:4892 , piece_id=FXH2ZZHACDLSQCGJ4NBVVGLYBDDKA5J7KXT4RKUWJSBR2FOT7JUA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.123127372 (13h33m55.638229942s) [noise=true , address=104.36.82.47:4520 , piece_id=TS2GSEHDUCPNVAJJGGO4HBJQYAY4MUWN2LK3B7EFI5IJ7BHUP5HQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.125615705 (13h33m55.640718274s) [noise=true , address=104.36.82.60:4365 , piece_id=F5LBKY4MEJI5AA26CHAHYT7D47B5SJU2SVBVHUMKRPKOH2YGLFRQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.129141651 (13h33m55.644244214s) [noise=true , address=104.36.82.60:4357 , piece_id=UTRV3ECM4VKIMVI25QOD3GA7MLBO247ZDXYL7ZOPLOMM2RLSU6VQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.135781424 (13h33m55.650883986s) [noise=true , address=104.36.82.5:4844 , piece_id=REIROBOO56LH63HN6GTX237VQ4KYEACO2LXJGQ4KWPC2VQVEIPFA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145947229 (13h33m55.661049806s) [noise=true , address=104.36.82.12:4167 , piece_id=WJNPLTBMBLD7Z2ECP5SSJPDTHH7TX6U24T2OKTS7NUYIYN3C3C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145973097 (13h33m55.661075674s) [seg_pos={0 0}, upload=13963]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:21.146446342 (13h33m55.661548919s) [noise=true , address=104.36.82.39:4752 , piece_id=XXGMFHJX2IRCNBOYINHMKCJT5YS2Y72W6PL2DJQORRSHJ4OYFBCQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection Aug 7 23:13:21.146527095 (13h33m55.661629665s) [noise=true , address=79.127.205.233:4629 , piece_id=JO4HOXAE4TUR44R55JVMESPMDHZVPRZUB2UI4RSPBXZWYPWUYO6Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fCtaroi1DDvaKo4kVVMLPpwDiLWwz377LEs7pyh3BHhvkiifX): protocol: read tcp 159.223.143.39:50930->79.127.205.233:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146720823 (13h33m55.661823393s) [noise=true , address=104.36.82.9:4009 , piece_id=WSQW7NZSRPNYA4AM4T2A5T6KUWEHJJ47E6D66Y4S76RIX2KY7TMQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb): protocol: read tcp 159.223.143.39:49842->104.36.82.9:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146939426 (13h33m55.662041993s) [noise=true , address=64.49.13.154:4936 , piece_id=CP354MRPGNHBGBGMDDXCQWV4HMXLDE2BX5N3SDRI25757TH7LPJA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J4nbkWW5oM17EbRwB53vgPzEiK8dEXM2tR2TWGhJFqEja9RHi): protocol: read tcp 159.223.143.39:39690->64.49.13.154:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:13:21.146954418 (13h33m55.662056994s) [seg_pos={0 0}, upload=13963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:21.147046940 (13h33m55.662149506s) [upload=13963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:21.147051626 (13h33m55.662154193s) [upload=13963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:21.197764982 (13h33m55.712867549s) [upload=13963]: Done uploading segments... Aug 7 23:13:21.197785171 (13h33m55.712887737s) [upload=13963]: Upload finished. err: <nil> Aug 7 23:13:52.912177255 (13h34m27.427279825s) [upload=13973]: Starting upload Aug 7 23:13:52.912245188 (13h34m27.427347764s) [upload=13973]: Done starting upload Aug 7 23:13:52.912322905 (13h34m27.427425471s) [upload=13973]: Uploading segments... Aug 7 23:13:52.912741963 (13h34m27.427844541s) [upload=13973]: Got next segment. Inline: false Aug 7 23:13:52.912787686 (13h34m27.427890271s) [upload=13973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:52.912794152 (13h34m27.427896731s) [upload=13973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:52.879876232 (13h35m27.394978808s) [upload=13973]: Next segment err: context deadline exceeded Aug 7 23:14:52.879935785 (13h35m27.395038353s) [upload=13973]: Done uploading segments... Aug 7 23:14:52.879952894 (13h35m27.395055462s) [upload=13973]: Upload finished. err: context deadline exceeded Aug 7 23:14:56.273083754 (13h35m30.78818632s) [upload=13987]: Starting upload Aug 7 23:14:56.273138206 (13h35m30.788240791s) [upload=13987]: Done starting upload Aug 7 23:14:56.273487156 (13h35m30.788589726s) [upload=13987]: Uploading segments... Aug 7 23:14:56.273528218 (13h35m30.788630804s) [upload=13987]: Got next segment. Inline: false Aug 7 23:14:56.273572431 (13h35m30.788675008s) [upload=13987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.273577073 (13h35m30.788679642s) [upload=13987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:56.244079248 (13h36m30.759181825s) [upload=13987]: Next segment err: context deadline exceeded Aug 7 23:15:56.244117923 (13h36m30.7592205s) [upload=13987]: Done uploading segments... Aug 7 23:15:56.244134836 (13h36m30.759237404s) [upload=13987]: Upload finished. err: context deadline exceeded Aug 7 23:15:56.354114004 (13h36m30.869216575s) [upload=13999]: Starting upload Aug 7 23:15:56.354173464 (13h36m30.869276042s) [upload=13999]: Done starting upload Aug 7 23:15:56.354235645 (13h36m30.869338234s) [upload=13999]: Uploading segments... Aug 7 23:15:56.354444202 (13h36m30.869546778s) [upload=13999]: Got next segment. Inline: false Aug 7 23:15:56.354481122 (13h36m30.869583698s) [upload=13999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:56.354484796 (13h36m30.869587372s) [upload=13999]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:56.323532882 (13h37m30.838635479s) [upload=13999]: Next segment err: context deadline exceeded Aug 7 23:16:56.323738594 (13h37m30.838841171s) [upload=13999]: Done uploading segments... Aug 7 23:16:56.323774957 (13h37m30.838877535s) [upload=13999]: Upload finished. err: context deadline exceeded Aug 7 23:17:01.850987007 (13h37m36.366089575s) [upload=14013]: Starting upload Aug 7 23:17:01.851135604 (13h37m36.366238172s) [upload=14013]: Done starting upload Aug 7 23:17:01.851286857 (13h37m36.366389428s) [upload=14013]: Uploading segments... Aug 7 23:17:01.851780544 (13h37m36.366883126s) [upload=14013]: Got next segment. Inline: false Aug 7 23:17:01.851834378 (13h37m36.366936962s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:01.851851393 (13h37m36.366953977s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:01.818716518 (13h38m36.333819106s) [upload=14013]: Next segment err: context deadline exceeded Aug 7 23:18:01.818844791 (13h38m36.333947369s) [upload=14013]: Done uploading segments... Aug 7 23:18:01.818870437 (13h38m36.33397301s) [upload=14013]: Upload finished. err: context deadline exceeded Duration: 484.858843ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..................................................X.XXX.XX..] duration: 9.282622188s FAIL (2025-08-07 23:19:12 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.282645618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XX.XXX...] duration: 286.746974ms FAIL (2025-08-07 23:19:08 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30m31.135428317s) [upload=13902]: Starting upload Aug 7 23:09:56.620515140 (13h30m31.135617722s) [upload=13902]: Done starting upload Aug 7 23:09:56.620600541 (13h30m31.135703106s) [upload=13902]: Uploading segments... Aug 7 23:09:56.620663979 (13h30m31.135766559s) [upload=13902]: Got next segment. Inline: true Aug 7 23:09:56.620669313 (13h30m31.13577189s) [upload=13902]: Waiting for error group managing segments... Aug 7 23:09:56.620725144 (13h30m31.13582772s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:56.620729700 (13h30m31.135832276s) [upload=13902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:56.620732858 (13h30m31.135835435s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:56.756931332 (13h30m31.272033912s) [upload=13902]: Done uploading segments... Aug 7 23:09:56.756952203 (13h30m31.272054766s) [upload=13902]: Upload finished. err: <nil> Aug 7 23:10:22.255312070 (13h30m56.770414636s) [upload=13913]: Starting upload Aug 7 23:10:22.255389033 (13h30m56.770491609s) [upload=13913]: Done starting upload Aug 7 23:10:22.255538785 (13h30m56.77064136s) [upload=13913]: Uploading segments... Aug 7 23:10:22.255632108 (13h30m56.77073469s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:22.255639107 (13h30m56.770741683s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:22.255686584 (13h30m56.77078915s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:22.255691754 (13h30m56.770794331s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:22.255695470 (13h30m56.770798046s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:22.322045125 (13h30m56.837147702s) [upload=13913]: Done uploading segments... Aug 7 23:10:22.322068753 (13h30m56.837171329s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:00.342239204 (13h31m34.85734177s) [upload=13922]: Starting upload Aug 7 23:11:00.342310667 (13h31m34.857413235s) [upload=13922]: Done starting upload Aug 7 23:11:00.342383830 (13h31m34.85748641s) [upload=13922]: Uploading segments... Aug 7 23:11:00.342449267 (13h31m34.857551854s) [upload=13922]: Got next segment. Inline: true Aug 7 23:11:00.342455504 (13h31m34.857558092s) [upload=13922]: Waiting for error group managing segments... Aug 7 23:11:00.342503627 (13h31m34.857606209s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:00.342508465 (13h31m34.857611046s) [upload=13922]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:00.342511859 (13h31m34.857614451s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:00.401913127 (13h31m34.917015712s) [upload=13922]: Done uploading segments... Aug 7 23:11:00.401931904 (13h31m34.917034472s) [upload=13922]: Upload finished. err: <nil> Aug 7 23:11:35.523175401 (13h32m10.03827798s) [upload=13934]: Starting upload Aug 7 23:11:35.523266566 (13h32m10.038369147s) [upload=13934]: Done starting upload Aug 7 23:11:35.523355068 (13h32m10.038457647s) [upload=13934]: Uploading segments... Aug 7 23:11:35.523405664 (13h32m10.03850825s) [upload=13934]: Got next segment. Inline: true Aug 7 23:11:35.523412393 (13h32m10.038514977s) [upload=13934]: Waiting for error group managing segments... Aug 7 23:11:35.523470368 (13h32m10.038572947s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:35.523479499 (13h32m10.038582083s) [upload=13934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:35.523484240 (13h32m10.038586821s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:35.587656651 (13h32m10.102759216s) [upload=13934]: Done uploading segments... Aug 7 23:11:35.587674771 (13h32m10.102777338s) [upload=13934]: Upload finished. err: <nil> Aug 7 23:12:23.226340957 (13h32m57.741443533s) [upload=13947]: Starting upload Aug 7 23:12:23.226411041 (13h32m57.741513618s) [upload=13947]: Done starting upload Aug 7 23:12:23.226470575 (13h32m57.74157314s) [upload=13947]: Uploading segments... Aug 7 23:12:23.226506745 (13h32m57.741609326s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:23.226511437 (13h32m57.741614019s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:23.226561258 (13h32m57.741663839s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:23.226566182 (13h32m57.741668762s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:23.226569380 (13h32m57.741671956s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:53.198277272 (13h33m27.713379848s) [upload=13947]: Done uploading segments... Aug 7 23:12:53.198295252 (13h33m27.713397814s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:53.310207725 (13h33m27.825310302s) [upload=13957]: Starting upload Aug 7 23:12:53.310290257 (13h33m27.82539282s) [upload=13957]: Done starting upload Aug 7 23:12:53.310379699 (13h33m27.825482276s) [upload=13957]: Uploading segments... Aug 7 23:12:53.310488747 (13h33m27.825591336s) [upload=13957]: Got next segment. Inline: true Aug 7 23:12:53.310497967 (13h33m27.825600548s) [upload=13957]: Waiting for error group managing segments... Aug 7 23:12:53.310564975 (13h33m27.825667562s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:53.310572430 (13h33m27.825675017s) [upload=13957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:53.310576051 (13h33m27.825678628s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:23.280527765 (13h33m57.79563034s) [upload=13957]: Done uploading segments... Aug 7 23:13:23.280561185 (13h33m57.795663762s) [upload=13957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:50.810735096 (13h34m25.325837673s) [upload=13972]: Starting upload Aug 7 23:13:50.810789852 (13h34m25.325892437s) [upload=13972]: Done starting upload Aug 7 23:13:50.810850987 (13h34m25.325953556s) [upload=13972]: Uploading segments... Aug 7 23:13:50.810923216 (13h34m25.326025797s) [upload=13972]: Got next segment. Inline: true Aug 7 23:13:50.810928523 (13h34m25.326031096s) [upload=13972]: Waiting for error group managing segments... Aug 7 23:13:50.810970440 (13h34m25.326073016s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:50.810974710 (13h34m25.326077286s) [upload=13972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:50.810978274 (13h34m25.32608085s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:50.866577223 (13h34m25.3816798s) [upload=13972]: Done uploading segments... Aug 7 23:13:50.866594654 (13h34m25.38169723s) [upload=13972]: Upload finished. err: <nil> Aug 7 23:14:54.225692687 (13h35m28.740795254s) [upload=13986]: Starting upload Aug 7 23:14:54.225785134 (13h35m28.740887721s) [upload=13986]: Done starting upload Aug 7 23:14:54.225870963 (13h35m28.740973529s) [upload=13986]: Uploading segments... Aug 7 23:14:54.225923809 (13h35m28.741026386s) [upload=13986]: Got next segment. Inline: true Aug 7 23:14:54.225928631 (13h35m28.741031207s) [upload=13986]: Waiting for error group managing segments... Aug 7 23:14:54.225969273 (13h35m28.741071842s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:54.225973734 (13h35m28.7410763s) [upload=13986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:54.225976991 (13h35m28.741079566s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:24.191802298 (13h35m58.706904867s) [upload=13986]: Done uploading segments... Aug 7 23:15:24.191822370 (13h35m58.70692494s) [upload=13986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:24.446619359 (13h35m58.961721924s) [upload=13992]: Starting upload Aug 7 23:15:24.446725754 (13h35m58.961828338s) [upload=13992]: Done starting upload Aug 7 23:15:24.446812887 (13h35m58.961915463s) [upload=13992]: Uploading segments... Aug 7 23:15:24.446871932 (13h35m58.961974519s) [upload=13992]: Got next segment. Inline: true Aug 7 23:15:24.446878746 (13h35m58.961981315s) [upload=13992]: Waiting for error group managing segments... Aug 7 23:15:24.446939653 (13h35m58.962042222s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:24.446945288 (13h35m58.962047863s) [upload=13992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:24.446950467 (13h35m58.96205303s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:54.414392139 (13h36m28.929494705s) [upload=13992]: Done uploading segments... Aug 7 23:15:54.414412085 (13h36m28.929514651s) [upload=13992]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:32.150263694 (13h37m6.665366261s) [upload=14009]: Starting upload Aug 7 23:16:32.150345034 (13h37m6.665447612s) [upload=14009]: Done starting upload Aug 7 23:16:32.150402507 (13h37m6.665505071s) [upload=14009]: Uploading segments... Aug 7 23:16:32.150453597 (13h37m6.665556176s) [upload=14009]: Got next segment. Inline: true Aug 7 23:16:32.150458385 (13h37m6.665560958s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:16:32.150501065 (13h37m6.665603641s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:32.150505833 (13h37m6.66560841s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:32.150509597 (13h37m6.665612174s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:02.044102103 (13h37m36.55920467s) [upload=14009]: Done uploading segments... Aug 7 23:17:02.044123177 (13h37m36.559225743s) [upload=14009]: Upload finished. err: metaclient: context deadline exceeded Duration: 286.778196ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XX.X.X..] duration: 1.483304754s FAIL (2025-08-07 23:19:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.48332797s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 695.259127ms FAIL (2025-08-07 23:19:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== g_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214654258 (12h54m14.729756841s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214886232 (12h54m14.729988875s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215184221 (12h54m14.730286807s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215580311 (12h54m14.730682897s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215600089 (12h54m14.730702668s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216063638 (12h54m14.731166221s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216221465 (12h54m14.73132405s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216499611 (12h54m14.73160219s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216784173 (12h54m14.731886753s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216959396 (12h54m14.732061963s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217229150 (12h54m14.732331732s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217364444 (12h54m14.732467028s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217574188 (12h54m14.732676766s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.218700401 (12h54m14.733802981s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.219113148 (12h54m14.734215718s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.307711467 (12h54m14.82281403s) [upload=13273]: Next returned nil segment Aug 7 22:33:40.307743585 (12h54m14.822846164s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:33:40.321114866 (12h54m14.836217441s) [noise=true , address=45.91.116.9:4294 , piece_id=UMCVXJVJ7VWVLOXYVN4KSE5VHM2YXJWOD45364KGN2T62PGBYNOA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.321899105 (12h54m14.837001676s) [noise=true , address=45.91.116.4:4943 , piece_id=4U7YPCLZI7HIZGGBGVWIWJLBRNPGIYOVVUD2UEJJVCMZMC426JIA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.330294085 (12h54m14.845396652s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.331982437 (12h54m14.847085014s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.334537739 (12h54m14.849640313s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.335955579 (12h54m14.851058147s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.340231956 (12h54m14.855334526s) [noise=true , address=77.90.31.11:4559 , piece_id=JSL5C3XIXINYAVAVTQ7J3QD3WWRG6FKP4H4DGJLDOPWCGI7CNAFA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.349513208 (12h54m14.864615777s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.358627538 (12h54m14.873730105s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.436339846 (12h54m14.951442414s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.543248537 (12h54m15.058351106s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.650678621 (12h54m15.165781209s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.786445525 (12h54m15.301548105s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907165087 (12h54m15.422267656s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907202527 (12h54m15.422305104s) [seg_pos={0 0}, upload=13273]: Segment reached optimal threshold of 14 pieces. Aug 7 22:33:40.907372225 (12h54m15.422474801s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QrrGrMHYAjyYiTdTXDUy2uH1j2p1qtUVJaDtais2GowJqBPPu): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.907642442 (12h54m15.422745018s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T): protocol: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection Aug 7 22:33:40.907950578 (12h54m15.423053162s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection Aug 7 22:33:40.908057652 (12h54m15.423160235s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.908080526 (12h54m15.423183103s) [seg_pos={0 0}, upload=13273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:33:40.908212573 (12h54m15.423315155s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:33:40.908218832 (12h54m15.423321415s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:41.213299020 (12h54m15.728401607s) [upload=13273]: Done uploading segments... Aug 7 22:33:41.213326550 (12h54m15.728429128s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:34:48.031458911 (12h55m22.546561477s) [upload=13295]: Starting upload Aug 7 22:34:48.031533867 (12h55m22.546636441s) [upload=13295]: Done starting upload Aug 7 22:34:48.031664339 (12h55m22.546766908s) [upload=13295]: Uploading segments... Aug 7 22:34:48.031895444 (12h55m22.546998028s) [upload=13295]: Got next segment. Inline: false Aug 7 22:34:48.031982233 (12h55m22.547084799s) [upload=13295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:48.031988106 (12h55m22.547090688s) [upload=13295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:34:48.040357077 (12h55m22.555459659s) [upload=13295]: Next segment err: metaclient: Too Many Requests Aug 7 22:34:48.040386485 (12h55m22.555489061s) [upload=13295]: Done uploading segments... Aug 7 22:34:48.040426787 (12h55m22.555529363s) [upload=13295]: Upload finished. err: metaclient: Too Many Requests Duration: 695.291033ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.753685611s FAIL (2025-08-07 23:18:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:11:04.814410393 (13h31m39.329512969s) [upload=13925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:11:04.814415016 (13h31m39.329517592s) [upload=13925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:04.947583472 (13h31m39.462686071s) [upload=13925]: Done uploading segments... Aug 7 23:11:04.947609736 (13h31m39.462712324s) [upload=13925]: Upload finished. err: <nil> Aug 7 23:11:29.227536921 (13h32m3.742639487s) [upload=13931]: Starting upload Aug 7 23:11:29.227600047 (13h32m3.742702623s) [upload=13931]: Done starting upload Aug 7 23:11:29.227715021 (13h32m3.742817591s) [upload=13931]: Uploading segments... Aug 7 23:11:29.227850209 (13h32m3.74295278s) [upload=13931]: Got next segment. Inline: false Aug 7 23:11:29.227898592 (13h32m3.743001169s) [upload=13931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:29.227903365 (13h32m3.743005942s) [upload=13931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:11:47.514990540 (13h32m22.03009312s) [seg_pos={0 0}, upload=13931]: Begin upload segment... Aug 7 23:11:47.515067094 (13h32m22.030169657s) [seg_pos={0 0}, upload=13931]: Done begin upload segment. Aug 7 23:11:47.515129228 (13h32m22.030231814s) [noise=true , address=77.90.31.18:4806 , piece_id=LI5HUM3HCYFYYSO6BENU4DQ2SFOPEJLRWZRM5PWCFXUZON4WCMAA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.515554241 (13h32m22.03065682s) [noise=true , address=104.36.82.14:4222 , piece_id=B5WB353FQ47THJP7MXV4AY5Z6FFSG7A66X72BLJUDK2ZTM3FNDVA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.516068554 (13h32m22.031171143s) [noise=true , address=45.91.116.7:4010 , piece_id=XOROULN3EEUSG45LZ5V53ORP2YV7CWLRQDSQ2AXREAFB62FGNL6A, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.516625763 (13h32m22.031728346s) [noise=true , address=45.91.116.8:4047 , piece_id=5E2VVSY65OAD6OFCQBBOTYLCLUKNVCVD3KY7OOVBI5TCYKQUKBQQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517060179 (13h32m22.03216276s) [noise=true , address=104.36.82.7:4910 , piece_id=Y75KUQR2IYRZBANBQPDFZTUJZPM54BA3KI6ZEF77NI3V755HPBOA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517396640 (13h32m22.032499217s) [noise=true , address=104.36.82.6:4873 , piece_id=3YGEWHQ7MXU6IPUJCYNM7L4DMNOECDSTVXMUBCX3CJCGPJ4OGGIQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517801490 (13h32m22.032904072s) [noise=true , address=45.91.116.10:4081 , piece_id=WDJ6KCU4LP75N7TSSR2VTWWEON3VRWE7BFJL57H2RM7ZYI2GYLWQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518135215 (13h32m22.033237797s) [noise=true , address=45.91.116.5:4297 , piece_id=T76WY44JQ55TJH4KJM57QWNBP7DVYNIBDVAPVN2AUTWASN4V76IA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518441967 (13h32m22.033544543s) [noise=true , address=77.90.31.11:4553 , piece_id=NK6AJDEDIAA5N3DNKBIPKZXURRSBOJG26FIMUB2SVFTIWOKFJWNQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518756911 (13h32m22.033859486s) [noise=true , address=77.90.31.21:4877 , piece_id=NVD7UBISRZJFOBJR7JKOX6V354Z6VCEP5TWJPGW3WOSI6S6K5FDA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519217404 (13h32m22.034319986s) [noise=true , address=45.91.116.6:4973 , piece_id=HEA5GQ53KZVIMBZ5LC3S4Y4372WMGUSATCBDTY5BIHZ4WL2PG7HQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519516685 (13h32m22.034619271s) [noise=true , address=77.90.31.19:4841 , piece_id=Z76CMQDB3DDLHQV35DZORZWRBX2D52BLBDCKPFXPONE3OUNGSGHA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519904148 (13h32m22.035006727s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7PUVWG4BXEJSS4YOBVTMUHCAWWKKQAR43NH4U3R24YMK3MP6H6Q, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.520220651 (13h32m22.035323239s) [noise=true , address=77.90.31.10:4517 , piece_id=WJ7TMWHMURKZ6HJON2NBIZVIC4XEXUUR5M7KD5PFWV4MEB3DUV7A, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.520644038 (13h32m22.035746619s) [noise=true , address=104.36.82.12:4149 , piece_id=UEFYG73QPD3OQX5IF2BGE7SQHYL5OKV37LQOKGNECBGRYQLULCJQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.521219880 (13h32m22.036322466s) [noise=true , address=77.90.31.17:4771 , piece_id=YCZ26U3QI4NQ7CKNVZEEB4O6XK4ZR3SPXTKT75P5U4ZQLUWOJWQA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.609483246 (13h32m22.124585826s) [upload=13931]: Next returned nil segment Aug 7 23:11:47.609501768 (13h32m22.124604348s) [upload=13931]: Waiting for error group managing segments... Aug 7 23:11:47.614240475 (13h32m22.129343046s) [noise=true , address=45.91.116.8:4047 , piece_id=5E2VVSY65OAD6OFCQBBOTYLCLUKNVCVD3KY7OOVBI5TCYKQUKBQQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.614925369 (13h32m22.130027939s) [noise=true , address=45.91.116.5:4297 , piece_id=T76WY44JQ55TJH4KJM57QWNBP7DVYNIBDVAPVN2AUTWASN4V76IA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.615059457 (13h32m22.130162026s) [noise=true , address=45.91.116.7:4010 , piece_id=XOROULN3EEUSG45LZ5V53ORP2YV7CWLRQDSQ2AXREAFB62FGNL6A, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.615458775 (13h32m22.130561338s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7PUVWG4BXEJSS4YOBVTMUHCAWWKKQAR43NH4U3R24YMK3MP6H6Q, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.616116276 (13h32m22.131218838s) [noise=true , address=77.90.31.18:4806 , piece_id=LI5HUM3HCYFYYSO6BENU4DQ2SFOPEJLRWZRM5PWCFXUZON4WCMAA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.616661348 (13h32m22.131763909s) [noise=true , address=77.90.31.17:4771 , piece_id=YCZ26U3QI4NQ7CKNVZEEB4O6XK4ZR3SPXTKT75P5U4ZQLUWOJWQA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.618000416 (13h32m22.133102978s) [noise=true , address=45.91.116.10:4081 , piece_id=WDJ6KCU4LP75N7TSSR2VTWWEON3VRWE7BFJL57H2RM7ZYI2GYLWQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.618530300 (13h32m22.133632866s) [noise=true , address=45.91.116.6:4973 , piece_id=HEA5GQ53KZVIMBZ5LC3S4Y4372WMGUSATCBDTY5BIHZ4WL2PG7HQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.619098772 (13h32m22.13420134s) [noise=true , address=77.90.31.19:4841 , piece_id=Z76CMQDB3DDLHQV35DZORZWRBX2D52BLBDCKPFXPONE3OUNGSGHA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.619697172 (13h32m22.13479975s) [noise=true , address=77.90.31.11:4553 , piece_id=NK6AJDEDIAA5N3DNKBIPKZXURRSBOJG26FIMUB2SVFTIWOKFJWNQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.620954558 (13h32m22.136057128s) [noise=true , address=77.90.31.21:4877 , piece_id=NVD7UBISRZJFOBJR7JKOX6V354Z6VCEP5TWJPGW3WOSI6S6K5FDA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.627010966 (13h32m22.142113543s) [noise=true , address=104.36.82.7:4910 , piece_id=Y75KUQR2IYRZBANBQPDFZTUJZPM54BA3KI6ZEF77NI3V755HPBOA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.629783113 (13h32m22.144885677s) [noise=true , address=77.90.31.10:4517 , piece_id=WJ7TMWHMURKZ6HJON2NBIZVIC4XEXUUR5M7KD5PFWV4MEB3DUV7A, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.629816031 (13h32m22.144918609s) [seg_pos={0 0}, upload=13931]: Segment reached optimal threshold of 13 pieces. Aug 7 23:11:47.630126623 (13h32m22.145229201s) [noise=true , address=104.36.82.14:4222 , piece_id=B5WB353FQ47THJP7MXV4AY5Z6FFSG7A66X72BLJUDK2ZTM3FNDVA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection Aug 7 23:11:47.630137290 (13h32m22.145239874s) [noise=true , address=104.36.82.12:4149 , piece_id=UEFYG73QPD3OQX5IF2BGE7SQHYL5OKV37LQOKGNECBGRYQLULCJQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection; write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection Aug 7 23:11:47.630232787 (13h32m22.145335361s) [noise=true , address=104.36.82.6:4873 , piece_id=3YGEWHQ7MXU6IPUJCYNM7L4DMNOECDSTVXMUBCX3CJCGPJ4OGGIQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 159.223.143.39:36062->104.36.82.6:4873: use of closed network connection; context canceled Aug 7 23:11:47.630264689 (13h32m22.145367273s) [seg_pos={0 0}, upload=13931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:11:47.630345104 (13h32m22.145447682s) [upload=13931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:11:47.630350066 (13h32m22.145452652s) [upload=13931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:37.334152859 (13h39m11.849255428s) [upload=13931]: Done uploading segments... Aug 7 23:18:37.334173589 (13h39m11.849276166s) [upload=13931]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.753706094s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.77562188s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:18:52 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 1.775637464s edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1m18.206155969s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:18:50 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 1m18.206176581s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...................................................X.XXX.XX.] duration: 1.175161621s FAIL (2025-08-07 23:18:45 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.175176826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....................................................X.XXX.XX] duration: 1m0.00102463s FAIL (2025-08-07 23:18:44 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001035495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 7m8.136402419s FAIL (2025-08-07 23:18:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:11:04.814410393 (13h31m39.329512969s) [upload=13925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:11:04.814415016 (13h31m39.329517592s) [upload=13925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:04.947583472 (13h31m39.462686071s) [upload=13925]: Done uploading segments... Aug 7 23:11:04.947609736 (13h31m39.462712324s) [upload=13925]: Upload finished. err: <nil> Aug 7 23:11:29.227536921 (13h32m3.742639487s) [upload=13931]: Starting upload Aug 7 23:11:29.227600047 (13h32m3.742702623s) [upload=13931]: Done starting upload Aug 7 23:11:29.227715021 (13h32m3.742817591s) [upload=13931]: Uploading segments... Aug 7 23:11:29.227850209 (13h32m3.74295278s) [upload=13931]: Got next segment. Inline: false Aug 7 23:11:29.227898592 (13h32m3.743001169s) [upload=13931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:29.227903365 (13h32m3.743005942s) [upload=13931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:11:47.514990540 (13h32m22.03009312s) [seg_pos={0 0}, upload=13931]: Begin upload segment... Aug 7 23:11:47.515067094 (13h32m22.030169657s) [seg_pos={0 0}, upload=13931]: Done begin upload segment. Aug 7 23:11:47.515129228 (13h32m22.030231814s) [noise=true , address=77.90.31.18:4806 , piece_id=LI5HUM3HCYFYYSO6BENU4DQ2SFOPEJLRWZRM5PWCFXUZON4WCMAA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.515554241 (13h32m22.03065682s) [noise=true , address=104.36.82.14:4222 , piece_id=B5WB353FQ47THJP7MXV4AY5Z6FFSG7A66X72BLJUDK2ZTM3FNDVA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.516068554 (13h32m22.031171143s) [noise=true , address=45.91.116.7:4010 , piece_id=XOROULN3EEUSG45LZ5V53ORP2YV7CWLRQDSQ2AXREAFB62FGNL6A, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.516625763 (13h32m22.031728346s) [noise=true , address=45.91.116.8:4047 , piece_id=5E2VVSY65OAD6OFCQBBOTYLCLUKNVCVD3KY7OOVBI5TCYKQUKBQQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517060179 (13h32m22.03216276s) [noise=true , address=104.36.82.7:4910 , piece_id=Y75KUQR2IYRZBANBQPDFZTUJZPM54BA3KI6ZEF77NI3V755HPBOA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517396640 (13h32m22.032499217s) [noise=true , address=104.36.82.6:4873 , piece_id=3YGEWHQ7MXU6IPUJCYNM7L4DMNOECDSTVXMUBCX3CJCGPJ4OGGIQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517801490 (13h32m22.032904072s) [noise=true , address=45.91.116.10:4081 , piece_id=WDJ6KCU4LP75N7TSSR2VTWWEON3VRWE7BFJL57H2RM7ZYI2GYLWQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518135215 (13h32m22.033237797s) [noise=true , address=45.91.116.5:4297 , piece_id=T76WY44JQ55TJH4KJM57QWNBP7DVYNIBDVAPVN2AUTWASN4V76IA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518441967 (13h32m22.033544543s) [noise=true , address=77.90.31.11:4553 , piece_id=NK6AJDEDIAA5N3DNKBIPKZXURRSBOJG26FIMUB2SVFTIWOKFJWNQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518756911 (13h32m22.033859486s) [noise=true , address=77.90.31.21:4877 , piece_id=NVD7UBISRZJFOBJR7JKOX6V354Z6VCEP5TWJPGW3WOSI6S6K5FDA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519217404 (13h32m22.034319986s) [noise=true , address=45.91.116.6:4973 , piece_id=HEA5GQ53KZVIMBZ5LC3S4Y4372WMGUSATCBDTY5BIHZ4WL2PG7HQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519516685 (13h32m22.034619271s) [noise=true , address=77.90.31.19:4841 , piece_id=Z76CMQDB3DDLHQV35DZORZWRBX2D52BLBDCKPFXPONE3OUNGSGHA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519904148 (13h32m22.035006727s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7PUVWG4BXEJSS4YOBVTMUHCAWWKKQAR43NH4U3R24YMK3MP6H6Q, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.520220651 (13h32m22.035323239s) [noise=true , address=77.90.31.10:4517 , piece_id=WJ7TMWHMURKZ6HJON2NBIZVIC4XEXUUR5M7KD5PFWV4MEB3DUV7A, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.520644038 (13h32m22.035746619s) [noise=true , address=104.36.82.12:4149 , piece_id=UEFYG73QPD3OQX5IF2BGE7SQHYL5OKV37LQOKGNECBGRYQLULCJQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.521219880 (13h32m22.036322466s) [noise=true , address=77.90.31.17:4771 , piece_id=YCZ26U3QI4NQ7CKNVZEEB4O6XK4ZR3SPXTKT75P5U4ZQLUWOJWQA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.609483246 (13h32m22.124585826s) [upload=13931]: Next returned nil segment Aug 7 23:11:47.609501768 (13h32m22.124604348s) [upload=13931]: Waiting for error group managing segments... Aug 7 23:11:47.614240475 (13h32m22.129343046s) [noise=true , address=45.91.116.8:4047 , piece_id=5E2VVSY65OAD6OFCQBBOTYLCLUKNVCVD3KY7OOVBI5TCYKQUKBQQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.614925369 (13h32m22.130027939s) [noise=true , address=45.91.116.5:4297 , piece_id=T76WY44JQ55TJH4KJM57QWNBP7DVYNIBDVAPVN2AUTWASN4V76IA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.615059457 (13h32m22.130162026s) [noise=true , address=45.91.116.7:4010 , piece_id=XOROULN3EEUSG45LZ5V53ORP2YV7CWLRQDSQ2AXREAFB62FGNL6A, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.615458775 (13h32m22.130561338s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7PUVWG4BXEJSS4YOBVTMUHCAWWKKQAR43NH4U3R24YMK3MP6H6Q, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.616116276 (13h32m22.131218838s) [noise=true , address=77.90.31.18:4806 , piece_id=LI5HUM3HCYFYYSO6BENU4DQ2SFOPEJLRWZRM5PWCFXUZON4WCMAA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.616661348 (13h32m22.131763909s) [noise=true , address=77.90.31.17:4771 , piece_id=YCZ26U3QI4NQ7CKNVZEEB4O6XK4ZR3SPXTKT75P5U4ZQLUWOJWQA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.618000416 (13h32m22.133102978s) [noise=true , address=45.91.116.10:4081 , piece_id=WDJ6KCU4LP75N7TSSR2VTWWEON3VRWE7BFJL57H2RM7ZYI2GYLWQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.618530300 (13h32m22.133632866s) [noise=true , address=45.91.116.6:4973 , piece_id=HEA5GQ53KZVIMBZ5LC3S4Y4372WMGUSATCBDTY5BIHZ4WL2PG7HQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.619098772 (13h32m22.13420134s) [noise=true , address=77.90.31.19:4841 , piece_id=Z76CMQDB3DDLHQV35DZORZWRBX2D52BLBDCKPFXPONE3OUNGSGHA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.619697172 (13h32m22.13479975s) [noise=true , address=77.90.31.11:4553 , piece_id=NK6AJDEDIAA5N3DNKBIPKZXURRSBOJG26FIMUB2SVFTIWOKFJWNQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.620954558 (13h32m22.136057128s) [noise=true , address=77.90.31.21:4877 , piece_id=NVD7UBISRZJFOBJR7JKOX6V354Z6VCEP5TWJPGW3WOSI6S6K5FDA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.627010966 (13h32m22.142113543s) [noise=true , address=104.36.82.7:4910 , piece_id=Y75KUQR2IYRZBANBQPDFZTUJZPM54BA3KI6ZEF77NI3V755HPBOA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.629783113 (13h32m22.144885677s) [noise=true , address=77.90.31.10:4517 , piece_id=WJ7TMWHMURKZ6HJON2NBIZVIC4XEXUUR5M7KD5PFWV4MEB3DUV7A, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.629816031 (13h32m22.144918609s) [seg_pos={0 0}, upload=13931]: Segment reached optimal threshold of 13 pieces. Aug 7 23:11:47.630126623 (13h32m22.145229201s) [noise=true , address=104.36.82.14:4222 , piece_id=B5WB353FQ47THJP7MXV4AY5Z6FFSG7A66X72BLJUDK2ZTM3FNDVA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection Aug 7 23:11:47.630137290 (13h32m22.145239874s) [noise=true , address=104.36.82.12:4149 , piece_id=UEFYG73QPD3OQX5IF2BGE7SQHYL5OKV37LQOKGNECBGRYQLULCJQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection; write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection Aug 7 23:11:47.630232787 (13h32m22.145335361s) [noise=true , address=104.36.82.6:4873 , piece_id=3YGEWHQ7MXU6IPUJCYNM7L4DMNOECDSTVXMUBCX3CJCGPJ4OGGIQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 159.223.143.39:36062->104.36.82.6:4873: use of closed network connection; context canceled Aug 7 23:11:47.630264689 (13h32m22.145367273s) [seg_pos={0 0}, upload=13931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:11:47.630345104 (13h32m22.145447682s) [upload=13931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:11:47.630350066 (13h32m22.145452652s) [upload=13931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:37.334152859 (13h39m11.849255428s) [upload=13931]: Done uploading segments... Aug 7 23:18:37.334173589 (13h39m11.849276166s) [upload=13931]: Upload finished. err: metaclient: context deadline exceeded Duration: 7m8.136427079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................................XX.XXXX..] duration: 1.276079234s FAIL (2025-08-07 23:18:30 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 783045 (13h33m55.532885624s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.018198306 (13h33m55.533300869s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.099506775 (13h33m55.614609341s) [upload=13963]: Next returned nil segment Aug 7 23:13:21.099521270 (13h33m55.614623846s) [upload=13963]: Waiting for error group managing segments... Aug 7 23:13:21.104019028 (13h33m55.619121597s) [noise=true , address=45.91.116.5:4305 , piece_id=V2T7C5TPEPR4KDVZI67JO2DIIQGXNTCLOBEOOIZYGQYN4TB5UCKQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.109200706 (13h33m55.624303283s) [noise=true , address=104.36.82.17:4352 , piece_id=2FNMB25WSQMNVTAP7UCNXFJQJLF7VXVWSVOQLN2CTF4N6GQUQAGA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.110092715 (13h33m55.625195286s) [noise=true , address=104.36.82.43:4837 , piece_id=TCE3IJZU7ZA4Z5ONPBDQANINXZXA3BHKORRZGBB7AN6WZULACQ5A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.111990406 (13h33m55.627092973s) [noise=true , address=104.36.82.62:4435 , piece_id=3XPVD4VZ43PTPELTHNJHSMYCD4CIZGG3DBDBXDG247J22HYT6O2Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.112988613 (13h33m55.628091177s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.113711920 (13h33m55.628814482s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114233050 (13h33m55.629335628s) [noise=true , address=104.36.82.38:4712 , piece_id=6G4TGYY3JILWXGGVFMLX3BO77QRHM4OXVXJNL3YMAD53EU55MZXA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114540807 (13h33m55.629643369s) [noise=true , address=45.91.116.8:4053 , piece_id=AB4MH3EBI2HTBZH6UZXJXHWPP7SK6TF6QAE6NHGVIJ363VOY3DVQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.122184649 (13h33m55.637287217s) [noise=true , address=104.36.82.44:4892 , piece_id=FXH2ZZHACDLSQCGJ4NBVVGLYBDDKA5J7KXT4RKUWJSBR2FOT7JUA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.123127372 (13h33m55.638229942s) [noise=true , address=104.36.82.47:4520 , piece_id=TS2GSEHDUCPNVAJJGGO4HBJQYAY4MUWN2LK3B7EFI5IJ7BHUP5HQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.125615705 (13h33m55.640718274s) [noise=true , address=104.36.82.60:4365 , piece_id=F5LBKY4MEJI5AA26CHAHYT7D47B5SJU2SVBVHUMKRPKOH2YGLFRQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.129141651 (13h33m55.644244214s) [noise=true , address=104.36.82.60:4357 , piece_id=UTRV3ECM4VKIMVI25QOD3GA7MLBO247ZDXYL7ZOPLOMM2RLSU6VQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.135781424 (13h33m55.650883986s) [noise=true , address=104.36.82.5:4844 , piece_id=REIROBOO56LH63HN6GTX237VQ4KYEACO2LXJGQ4KWPC2VQVEIPFA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145947229 (13h33m55.661049806s) [noise=true , address=104.36.82.12:4167 , piece_id=WJNPLTBMBLD7Z2ECP5SSJPDTHH7TX6U24T2OKTS7NUYIYN3C3C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145973097 (13h33m55.661075674s) [seg_pos={0 0}, upload=13963]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:21.146446342 (13h33m55.661548919s) [noise=true , address=104.36.82.39:4752 , piece_id=XXGMFHJX2IRCNBOYINHMKCJT5YS2Y72W6PL2DJQORRSHJ4OYFBCQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection Aug 7 23:13:21.146527095 (13h33m55.661629665s) [noise=true , address=79.127.205.233:4629 , piece_id=JO4HOXAE4TUR44R55JVMESPMDHZVPRZUB2UI4RSPBXZWYPWUYO6Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fCtaroi1DDvaKo4kVVMLPpwDiLWwz377LEs7pyh3BHhvkiifX): protocol: read tcp 159.223.143.39:50930->79.127.205.233:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146720823 (13h33m55.661823393s) [noise=true , address=104.36.82.9:4009 , piece_id=WSQW7NZSRPNYA4AM4T2A5T6KUWEHJJ47E6D66Y4S76RIX2KY7TMQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb): protocol: read tcp 159.223.143.39:49842->104.36.82.9:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146939426 (13h33m55.662041993s) [noise=true , address=64.49.13.154:4936 , piece_id=CP354MRPGNHBGBGMDDXCQWV4HMXLDE2BX5N3SDRI25757TH7LPJA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J4nbkWW5oM17EbRwB53vgPzEiK8dEXM2tR2TWGhJFqEja9RHi): protocol: read tcp 159.223.143.39:39690->64.49.13.154:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:13:21.146954418 (13h33m55.662056994s) [seg_pos={0 0}, upload=13963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:21.147046940 (13h33m55.662149506s) [upload=13963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:21.147051626 (13h33m55.662154193s) [upload=13963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:21.197764982 (13h33m55.712867549s) [upload=13963]: Done uploading segments... Aug 7 23:13:21.197785171 (13h33m55.712887737s) [upload=13963]: Upload finished. err: <nil> Aug 7 23:13:52.912177255 (13h34m27.427279825s) [upload=13973]: Starting upload Aug 7 23:13:52.912245188 (13h34m27.427347764s) [upload=13973]: Done starting upload Aug 7 23:13:52.912322905 (13h34m27.427425471s) [upload=13973]: Uploading segments... Aug 7 23:13:52.912741963 (13h34m27.427844541s) [upload=13973]: Got next segment. Inline: false Aug 7 23:13:52.912787686 (13h34m27.427890271s) [upload=13973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:52.912794152 (13h34m27.427896731s) [upload=13973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:52.879876232 (13h35m27.394978808s) [upload=13973]: Next segment err: context deadline exceeded Aug 7 23:14:52.879935785 (13h35m27.395038353s) [upload=13973]: Done uploading segments... Aug 7 23:14:52.879952894 (13h35m27.395055462s) [upload=13973]: Upload finished. err: context deadline exceeded Aug 7 23:14:56.273083754 (13h35m30.78818632s) [upload=13987]: Starting upload Aug 7 23:14:56.273138206 (13h35m30.788240791s) [upload=13987]: Done starting upload Aug 7 23:14:56.273487156 (13h35m30.788589726s) [upload=13987]: Uploading segments... Aug 7 23:14:56.273528218 (13h35m30.788630804s) [upload=13987]: Got next segment. Inline: false Aug 7 23:14:56.273572431 (13h35m30.788675008s) [upload=13987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.273577073 (13h35m30.788679642s) [upload=13987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:56.244079248 (13h36m30.759181825s) [upload=13987]: Next segment err: context deadline exceeded Aug 7 23:15:56.244117923 (13h36m30.7592205s) [upload=13987]: Done uploading segments... Aug 7 23:15:56.244134836 (13h36m30.759237404s) [upload=13987]: Upload finished. err: context deadline exceeded Aug 7 23:15:56.354114004 (13h36m30.869216575s) [upload=13999]: Starting upload Aug 7 23:15:56.354173464 (13h36m30.869276042s) [upload=13999]: Done starting upload Aug 7 23:15:56.354235645 (13h36m30.869338234s) [upload=13999]: Uploading segments... Aug 7 23:15:56.354444202 (13h36m30.869546778s) [upload=13999]: Got next segment. Inline: false Aug 7 23:15:56.354481122 (13h36m30.869583698s) [upload=13999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:56.354484796 (13h36m30.869587372s) [upload=13999]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:56.323532882 (13h37m30.838635479s) [upload=13999]: Next segment err: context deadline exceeded Aug 7 23:16:56.323738594 (13h37m30.838841171s) [upload=13999]: Done uploading segments... Aug 7 23:16:56.323774957 (13h37m30.838877535s) [upload=13999]: Upload finished. err: context deadline exceeded Aug 7 23:17:01.850987007 (13h37m36.366089575s) [upload=14013]: Starting upload Aug 7 23:17:01.851135604 (13h37m36.366238172s) [upload=14013]: Done starting upload Aug 7 23:17:01.851286857 (13h37m36.366389428s) [upload=14013]: Uploading segments... Aug 7 23:17:01.851780544 (13h37m36.366883126s) [upload=14013]: Got next segment. Inline: false Aug 7 23:17:01.851834378 (13h37m36.366936962s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:01.851851393 (13h37m36.366953977s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:01.818716518 (13h38m36.333819106s) [upload=14013]: Next segment err: context deadline exceeded Aug 7 23:18:01.818844791 (13h38m36.333947369s) [upload=14013]: Done uploading segments... Aug 7 23:18:01.818870437 (13h38m36.33397301s) [upload=14013]: Upload finished. err: context deadline exceeded Duration: 1.276105154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration: 2.28425993s FAIL (2025-08-07 23:18:25 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.284291947s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 26.118524ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:18:23 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 26.142937ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXX..X.] duration: 1.036042135s FAIL (2025-08-07 23:18:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.036133817s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XX..X..] duration: 664.472561ms FAIL (2025-08-07 23:18:11 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad finished. err: <nil> Aug 7 23:07:56.297738414 (13h28m30.812840994s) [upload=13869]: Starting upload Aug 7 23:07:56.297858483 (13h28m30.812961062s) [upload=13869]: Done starting upload Aug 7 23:07:56.297993444 (13h28m30.813096021s) [upload=13869]: Uploading segments... Aug 7 23:07:56.298078193 (13h28m30.813180778s) [upload=13869]: Got next segment. Inline: true Aug 7 23:07:56.298102093 (13h28m30.813204678s) [upload=13869]: Waiting for error group managing segments... Aug 7 23:07:56.298156325 (13h28m30.813258907s) [upload=13869]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:56.298177912 (13h28m30.813280494s) [upload=13869]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:56.298182546 (13h28m30.813285127s) [upload=13869]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:56.372043207 (13h28m30.887145784s) [upload=13869]: Done uploading segments... Aug 7 23:07:56.372061460 (13h28m30.887164036s) [upload=13869]: Upload finished. err: <nil> Aug 7 23:08:37.049508399 (13h29m11.564610963s) [upload=13879]: Starting upload Aug 7 23:08:37.049588017 (13h29m11.564690596s) [upload=13879]: Done starting upload Aug 7 23:08:37.049683838 (13h29m11.564786416s) [upload=13879]: Uploading segments... Aug 7 23:08:37.049829834 (13h29m11.564932424s) [upload=13879]: Got next segment. Inline: true Aug 7 23:08:37.049839426 (13h29m11.564942003s) [upload=13879]: Waiting for error group managing segments... Aug 7 23:08:37.049910072 (13h29m11.565012644s) [upload=13879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:37.049916484 (13h29m11.56501906s) [upload=13879]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:37.049920022 (13h29m11.565022598s) [upload=13879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:37.140370137 (13h29m11.655472717s) [upload=13879]: Done uploading segments... Aug 7 23:08:37.141250272 (13h29m11.65635286s) [upload=13879]: Upload finished. err: <nil> Aug 7 23:09:29.216369700 (13h30m3.731472269s) [upload=13895]: Starting upload Aug 7 23:09:29.216489949 (13h30m3.731592525s) [upload=13895]: Done starting upload Aug 7 23:09:29.216567331 (13h30m3.731669896s) [upload=13895]: Uploading segments... Aug 7 23:09:29.216612331 (13h30m3.731714917s) [upload=13895]: Got next segment. Inline: true Aug 7 23:09:29.216618125 (13h30m3.731720692s) [upload=13895]: Waiting for error group managing segments... Aug 7 23:09:29.216659267 (13h30m3.731761843s) [upload=13895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:29.216663938 (13h30m3.731766517s) [upload=13895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:29.216668553 (13h30m3.731771139s) [upload=13895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:29.297928170 (13h30m3.813030739s) [upload=13895]: Done uploading segments... Aug 7 23:09:29.297957487 (13h30m3.813060063s) [upload=13895]: Upload finished. err: <nil> Aug 7 23:10:23.115383547 (13h30m57.630486132s) [upload=13915]: Starting upload Aug 7 23:10:23.115456203 (13h30m57.630558786s) [upload=13915]: Done starting upload Aug 7 23:10:23.115557871 (13h30m57.630660452s) [upload=13915]: Uploading segments... Aug 7 23:10:23.115614526 (13h30m57.630717116s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:23.115621076 (13h30m57.630723665s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:23.115685040 (13h30m57.630787627s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:23.115692213 (13h30m57.630794802s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:23.115697119 (13h30m57.630799709s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:23.183146377 (13h30m57.698248962s) [upload=13915]: Done uploading segments... Aug 7 23:10:23.183169407 (13h30m57.698271988s) [upload=13915]: Upload finished. err: <nil> Aug 7 23:10:46.162725354 (13h31m20.67782793s) [upload=13919]: Starting upload Aug 7 23:10:46.162807728 (13h31m20.677910304s) [upload=13919]: Done starting upload Aug 7 23:10:46.162877771 (13h31m20.677980347s) [upload=13919]: Uploading segments... Aug 7 23:10:46.162939319 (13h31m20.678041901s) [upload=13919]: Got next segment. Inline: true Aug 7 23:10:46.162944044 (13h31m20.678046624s) [upload=13919]: Waiting for error group managing segments... Aug 7 23:10:46.162986704 (13h31m20.678089286s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:46.162991049 (13h31m20.678093629s) [upload=13919]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:46.162994213 (13h31m20.678096792s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:49.331481110 (13h31m23.84658369s) [upload=13919]: Done uploading segments... Aug 7 23:10:49.331502710 (13h31m23.846605285s) [upload=13919]: Upload finished. err: <nil> Aug 7 23:11:56.588799333 (13h32m31.103901909s) [upload=13942]: Starting upload Aug 7 23:11:56.588871172 (13h32m31.103973747s) [upload=13942]: Done starting upload Aug 7 23:11:56.588959193 (13h32m31.104061755s) [upload=13942]: Uploading segments... Aug 7 23:11:56.588996486 (13h32m31.104099072s) [upload=13942]: Got next segment. Inline: true Aug 7 23:11:56.589000914 (13h32m31.104103491s) [upload=13942]: Waiting for error group managing segments... Aug 7 23:11:56.589047870 (13h32m31.104150433s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.589051856 (13h32m31.104154432s) [upload=13942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.589054372 (13h32m31.10415695s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:56.555383755 (13h33m31.070486344s) [upload=13942]: Done uploading segments... Aug 7 23:12:56.555407795 (13h33m31.070510378s) [upload=13942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:56.669023639 (13h33m31.184126201s) [upload=13959]: Starting upload Aug 7 23:12:56.669081512 (13h33m31.184184088s) [upload=13959]: Done starting upload Aug 7 23:12:56.669139265 (13h33m31.184241827s) [upload=13959]: Uploading segments... Aug 7 23:12:56.669181725 (13h33m31.184284307s) [upload=13959]: Got next segment. Inline: true Aug 7 23:12:56.669185727 (13h33m31.18428829s) [upload=13959]: Waiting for error group managing segments... Aug 7 23:12:56.669222629 (13h33m31.184325191s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:56.669226085 (13h33m31.18432866s) [upload=13959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:56.669228556 (13h33m31.184331118s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.639927224 (13h34m31.155029791s) [upload=13959]: Done uploading segments... Aug 7 23:13:56.639946863 (13h34m31.155049429s) [upload=13959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:56.754611797 (13h34m31.269714367s) [upload=13974]: Starting upload Aug 7 23:13:56.754690544 (13h34m31.269793123s) [upload=13974]: Done starting upload Aug 7 23:13:56.754787910 (13h34m31.269890478s) [upload=13974]: Uploading segments... Aug 7 23:13:56.754829279 (13h34m31.269931864s) [upload=13974]: Got next segment. Inline: true Aug 7 23:13:56.754833834 (13h34m31.269936411s) [upload=13974]: Waiting for error group managing segments... Aug 7 23:13:56.754887650 (13h34m31.269990217s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.754891993 (13h34m31.26999456s) [upload=13974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.754895495 (13h34m31.269998071s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.814228870 (13h34m31.32933145s) [upload=13974]: Done uploading segments... Aug 7 23:13:56.814253732 (13h34m31.329356295s) [upload=13974]: Upload finished. err: <nil> Aug 7 23:14:51.237083244 (13h35m25.75218581s) [upload=13984]: Starting upload Aug 7 23:14:51.237154440 (13h35m25.752257016s) [upload=13984]: Done starting upload Aug 7 23:14:51.237222602 (13h35m25.752325168s) [upload=13984]: Uploading segments... Aug 7 23:14:51.237272573 (13h35m25.752375158s) [upload=13984]: Got next segment. Inline: true Aug 7 23:14:51.237277088 (13h35m25.752379654s) [upload=13984]: Waiting for error group managing segments... Aug 7 23:14:51.237328372 (13h35m25.752430937s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:51.237332974 (13h35m25.75243555s) [upload=13984]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:51.237336117 (13h35m25.752438684s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:51.367347705 (13h35m25.882450281s) [upload=13984]: Done uploading segments... Aug 7 23:14:51.367368302 (13h35m25.882470865s) [upload=13984]: Upload finished. err: <nil> Aug 7 23:16:05.237199068 (13h36m39.752301655s) [upload=14002]: Starting upload Aug 7 23:16:05.237269888 (13h36m39.752372457s) [upload=14002]: Done starting upload Aug 7 23:16:05.237360382 (13h36m39.752462952s) [upload=14002]: Uploading segments... Aug 7 23:16:05.237841750 (13h36m39.752944336s) [upload=14002]: Got next segment. Inline: true Aug 7 23:16:05.237859632 (13h36m39.752962218s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:16:05.237930004 (13h36m39.753032586s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.237936533 (13h36m39.753039109s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.237940573 (13h36m39.753043148s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.205359206 (13h37m39.720461782s) [upload=14002]: Done uploading segments... Aug 7 23:17:05.205376470 (13h37m39.720479046s) [upload=14002]: Upload finished. err: metaclient: context deadline exceeded Duration: 664.500868ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [...................................................X..XXXXX.] duration: 650.878018ms FAIL (2025-08-07 23:18:08 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 650.904826ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX..X] duration: 30.000959916s FAIL (2025-08-07 23:18:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000988295s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................................XX.XXXX.] duration: 1.139838295s FAIL (2025-08-07 23:18:03 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 783045 (13h33m55.532885624s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.018198306 (13h33m55.533300869s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.099506775 (13h33m55.614609341s) [upload=13963]: Next returned nil segment Aug 7 23:13:21.099521270 (13h33m55.614623846s) [upload=13963]: Waiting for error group managing segments... Aug 7 23:13:21.104019028 (13h33m55.619121597s) [noise=true , address=45.91.116.5:4305 , piece_id=V2T7C5TPEPR4KDVZI67JO2DIIQGXNTCLOBEOOIZYGQYN4TB5UCKQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.109200706 (13h33m55.624303283s) [noise=true , address=104.36.82.17:4352 , piece_id=2FNMB25WSQMNVTAP7UCNXFJQJLF7VXVWSVOQLN2CTF4N6GQUQAGA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.110092715 (13h33m55.625195286s) [noise=true , address=104.36.82.43:4837 , piece_id=TCE3IJZU7ZA4Z5ONPBDQANINXZXA3BHKORRZGBB7AN6WZULACQ5A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.111990406 (13h33m55.627092973s) [noise=true , address=104.36.82.62:4435 , piece_id=3XPVD4VZ43PTPELTHNJHSMYCD4CIZGG3DBDBXDG247J22HYT6O2Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.112988613 (13h33m55.628091177s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.113711920 (13h33m55.628814482s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114233050 (13h33m55.629335628s) [noise=true , address=104.36.82.38:4712 , piece_id=6G4TGYY3JILWXGGVFMLX3BO77QRHM4OXVXJNL3YMAD53EU55MZXA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114540807 (13h33m55.629643369s) [noise=true , address=45.91.116.8:4053 , piece_id=AB4MH3EBI2HTBZH6UZXJXHWPP7SK6TF6QAE6NHGVIJ363VOY3DVQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.122184649 (13h33m55.637287217s) [noise=true , address=104.36.82.44:4892 , piece_id=FXH2ZZHACDLSQCGJ4NBVVGLYBDDKA5J7KXT4RKUWJSBR2FOT7JUA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.123127372 (13h33m55.638229942s) [noise=true , address=104.36.82.47:4520 , piece_id=TS2GSEHDUCPNVAJJGGO4HBJQYAY4MUWN2LK3B7EFI5IJ7BHUP5HQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.125615705 (13h33m55.640718274s) [noise=true , address=104.36.82.60:4365 , piece_id=F5LBKY4MEJI5AA26CHAHYT7D47B5SJU2SVBVHUMKRPKOH2YGLFRQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.129141651 (13h33m55.644244214s) [noise=true , address=104.36.82.60:4357 , piece_id=UTRV3ECM4VKIMVI25QOD3GA7MLBO247ZDXYL7ZOPLOMM2RLSU6VQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.135781424 (13h33m55.650883986s) [noise=true , address=104.36.82.5:4844 , piece_id=REIROBOO56LH63HN6GTX237VQ4KYEACO2LXJGQ4KWPC2VQVEIPFA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145947229 (13h33m55.661049806s) [noise=true , address=104.36.82.12:4167 , piece_id=WJNPLTBMBLD7Z2ECP5SSJPDTHH7TX6U24T2OKTS7NUYIYN3C3C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145973097 (13h33m55.661075674s) [seg_pos={0 0}, upload=13963]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:21.146446342 (13h33m55.661548919s) [noise=true , address=104.36.82.39:4752 , piece_id=XXGMFHJX2IRCNBOYINHMKCJT5YS2Y72W6PL2DJQORRSHJ4OYFBCQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection Aug 7 23:13:21.146527095 (13h33m55.661629665s) [noise=true , address=79.127.205.233:4629 , piece_id=JO4HOXAE4TUR44R55JVMESPMDHZVPRZUB2UI4RSPBXZWYPWUYO6Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fCtaroi1DDvaKo4kVVMLPpwDiLWwz377LEs7pyh3BHhvkiifX): protocol: read tcp 159.223.143.39:50930->79.127.205.233:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146720823 (13h33m55.661823393s) [noise=true , address=104.36.82.9:4009 , piece_id=WSQW7NZSRPNYA4AM4T2A5T6KUWEHJJ47E6D66Y4S76RIX2KY7TMQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb): protocol: read tcp 159.223.143.39:49842->104.36.82.9:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146939426 (13h33m55.662041993s) [noise=true , address=64.49.13.154:4936 , piece_id=CP354MRPGNHBGBGMDDXCQWV4HMXLDE2BX5N3SDRI25757TH7LPJA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J4nbkWW5oM17EbRwB53vgPzEiK8dEXM2tR2TWGhJFqEja9RHi): protocol: read tcp 159.223.143.39:39690->64.49.13.154:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:13:21.146954418 (13h33m55.662056994s) [seg_pos={0 0}, upload=13963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:21.147046940 (13h33m55.662149506s) [upload=13963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:21.147051626 (13h33m55.662154193s) [upload=13963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:21.197764982 (13h33m55.712867549s) [upload=13963]: Done uploading segments... Aug 7 23:13:21.197785171 (13h33m55.712887737s) [upload=13963]: Upload finished. err: <nil> Aug 7 23:13:52.912177255 (13h34m27.427279825s) [upload=13973]: Starting upload Aug 7 23:13:52.912245188 (13h34m27.427347764s) [upload=13973]: Done starting upload Aug 7 23:13:52.912322905 (13h34m27.427425471s) [upload=13973]: Uploading segments... Aug 7 23:13:52.912741963 (13h34m27.427844541s) [upload=13973]: Got next segment. Inline: false Aug 7 23:13:52.912787686 (13h34m27.427890271s) [upload=13973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:52.912794152 (13h34m27.427896731s) [upload=13973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:52.879876232 (13h35m27.394978808s) [upload=13973]: Next segment err: context deadline exceeded Aug 7 23:14:52.879935785 (13h35m27.395038353s) [upload=13973]: Done uploading segments... Aug 7 23:14:52.879952894 (13h35m27.395055462s) [upload=13973]: Upload finished. err: context deadline exceeded Aug 7 23:14:56.273083754 (13h35m30.78818632s) [upload=13987]: Starting upload Aug 7 23:14:56.273138206 (13h35m30.788240791s) [upload=13987]: Done starting upload Aug 7 23:14:56.273487156 (13h35m30.788589726s) [upload=13987]: Uploading segments... Aug 7 23:14:56.273528218 (13h35m30.788630804s) [upload=13987]: Got next segment. Inline: false Aug 7 23:14:56.273572431 (13h35m30.788675008s) [upload=13987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.273577073 (13h35m30.788679642s) [upload=13987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:56.244079248 (13h36m30.759181825s) [upload=13987]: Next segment err: context deadline exceeded Aug 7 23:15:56.244117923 (13h36m30.7592205s) [upload=13987]: Done uploading segments... Aug 7 23:15:56.244134836 (13h36m30.759237404s) [upload=13987]: Upload finished. err: context deadline exceeded Aug 7 23:15:56.354114004 (13h36m30.869216575s) [upload=13999]: Starting upload Aug 7 23:15:56.354173464 (13h36m30.869276042s) [upload=13999]: Done starting upload Aug 7 23:15:56.354235645 (13h36m30.869338234s) [upload=13999]: Uploading segments... Aug 7 23:15:56.354444202 (13h36m30.869546778s) [upload=13999]: Got next segment. Inline: false Aug 7 23:15:56.354481122 (13h36m30.869583698s) [upload=13999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:56.354484796 (13h36m30.869587372s) [upload=13999]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:56.323532882 (13h37m30.838635479s) [upload=13999]: Next segment err: context deadline exceeded Aug 7 23:16:56.323738594 (13h37m30.838841171s) [upload=13999]: Done uploading segments... Aug 7 23:16:56.323774957 (13h37m30.838877535s) [upload=13999]: Upload finished. err: context deadline exceeded Aug 7 23:17:01.850987007 (13h37m36.366089575s) [upload=14013]: Starting upload Aug 7 23:17:01.851135604 (13h37m36.366238172s) [upload=14013]: Done starting upload Aug 7 23:17:01.851286857 (13h37m36.366389428s) [upload=14013]: Uploading segments... Aug 7 23:17:01.851780544 (13h37m36.366883126s) [upload=14013]: Got next segment. Inline: false Aug 7 23:17:01.851834378 (13h37m36.366936962s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:01.851851393 (13h37m36.366953977s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:01.818716518 (13h38m36.333819106s) [upload=14013]: Next segment err: context deadline exceeded Aug 7 23:18:01.818844791 (13h38m36.333947369s) [upload=14013]: Done uploading segments... Aug 7 23:18:01.818870437 (13h38m36.33397301s) [upload=14013]: Upload finished. err: context deadline exceeded Duration: 1.13986494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.588825059s FAIL (2025-08-07 23:18:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== g_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214654258 (12h54m14.729756841s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214886232 (12h54m14.729988875s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215184221 (12h54m14.730286807s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215580311 (12h54m14.730682897s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215600089 (12h54m14.730702668s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216063638 (12h54m14.731166221s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216221465 (12h54m14.73132405s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216499611 (12h54m14.73160219s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216784173 (12h54m14.731886753s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216959396 (12h54m14.732061963s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217229150 (12h54m14.732331732s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217364444 (12h54m14.732467028s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217574188 (12h54m14.732676766s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.218700401 (12h54m14.733802981s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.219113148 (12h54m14.734215718s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.307711467 (12h54m14.82281403s) [upload=13273]: Next returned nil segment Aug 7 22:33:40.307743585 (12h54m14.822846164s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:33:40.321114866 (12h54m14.836217441s) [noise=true , address=45.91.116.9:4294 , piece_id=UMCVXJVJ7VWVLOXYVN4KSE5VHM2YXJWOD45364KGN2T62PGBYNOA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.321899105 (12h54m14.837001676s) [noise=true , address=45.91.116.4:4943 , piece_id=4U7YPCLZI7HIZGGBGVWIWJLBRNPGIYOVVUD2UEJJVCMZMC426JIA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.330294085 (12h54m14.845396652s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.331982437 (12h54m14.847085014s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.334537739 (12h54m14.849640313s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.335955579 (12h54m14.851058147s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.340231956 (12h54m14.855334526s) [noise=true , address=77.90.31.11:4559 , piece_id=JSL5C3XIXINYAVAVTQ7J3QD3WWRG6FKP4H4DGJLDOPWCGI7CNAFA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.349513208 (12h54m14.864615777s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.358627538 (12h54m14.873730105s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.436339846 (12h54m14.951442414s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.543248537 (12h54m15.058351106s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.650678621 (12h54m15.165781209s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.786445525 (12h54m15.301548105s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907165087 (12h54m15.422267656s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907202527 (12h54m15.422305104s) [seg_pos={0 0}, upload=13273]: Segment reached optimal threshold of 14 pieces. Aug 7 22:33:40.907372225 (12h54m15.422474801s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QrrGrMHYAjyYiTdTXDUy2uH1j2p1qtUVJaDtais2GowJqBPPu): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.907642442 (12h54m15.422745018s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T): protocol: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection Aug 7 22:33:40.907950578 (12h54m15.423053162s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection Aug 7 22:33:40.908057652 (12h54m15.423160235s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.908080526 (12h54m15.423183103s) [seg_pos={0 0}, upload=13273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:33:40.908212573 (12h54m15.423315155s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:33:40.908218832 (12h54m15.423321415s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:41.213299020 (12h54m15.728401607s) [upload=13273]: Done uploading segments... Aug 7 22:33:41.213326550 (12h54m15.728429128s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:34:48.031458911 (12h55m22.546561477s) [upload=13295]: Starting upload Aug 7 22:34:48.031533867 (12h55m22.546636441s) [upload=13295]: Done starting upload Aug 7 22:34:48.031664339 (12h55m22.546766908s) [upload=13295]: Uploading segments... Aug 7 22:34:48.031895444 (12h55m22.546998028s) [upload=13295]: Got next segment. Inline: false Aug 7 22:34:48.031982233 (12h55m22.547084799s) [upload=13295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:48.031988106 (12h55m22.547090688s) [upload=13295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:34:48.040357077 (12h55m22.555459659s) [upload=13295]: Next segment err: metaclient: Too Many Requests Aug 7 22:34:48.040386485 (12h55m22.555489061s) [upload=13295]: Done uploading segments... Aug 7 22:34:48.040426787 (12h55m22.555529363s) [upload=13295]: Upload finished. err: metaclient: Too Many Requests Duration: 1.5888565s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XX.XXXX] duration: 1m0.001234954s FAIL (2025-08-07 23:18:01 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 783045 (13h33m55.532885624s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.018198306 (13h33m55.533300869s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.099506775 (13h33m55.614609341s) [upload=13963]: Next returned nil segment Aug 7 23:13:21.099521270 (13h33m55.614623846s) [upload=13963]: Waiting for error group managing segments... Aug 7 23:13:21.104019028 (13h33m55.619121597s) [noise=true , address=45.91.116.5:4305 , piece_id=V2T7C5TPEPR4KDVZI67JO2DIIQGXNTCLOBEOOIZYGQYN4TB5UCKQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.109200706 (13h33m55.624303283s) [noise=true , address=104.36.82.17:4352 , piece_id=2FNMB25WSQMNVTAP7UCNXFJQJLF7VXVWSVOQLN2CTF4N6GQUQAGA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.110092715 (13h33m55.625195286s) [noise=true , address=104.36.82.43:4837 , piece_id=TCE3IJZU7ZA4Z5ONPBDQANINXZXA3BHKORRZGBB7AN6WZULACQ5A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.111990406 (13h33m55.627092973s) [noise=true , address=104.36.82.62:4435 , piece_id=3XPVD4VZ43PTPELTHNJHSMYCD4CIZGG3DBDBXDG247J22HYT6O2Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.112988613 (13h33m55.628091177s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.113711920 (13h33m55.628814482s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114233050 (13h33m55.629335628s) [noise=true , address=104.36.82.38:4712 , piece_id=6G4TGYY3JILWXGGVFMLX3BO77QRHM4OXVXJNL3YMAD53EU55MZXA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114540807 (13h33m55.629643369s) [noise=true , address=45.91.116.8:4053 , piece_id=AB4MH3EBI2HTBZH6UZXJXHWPP7SK6TF6QAE6NHGVIJ363VOY3DVQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.122184649 (13h33m55.637287217s) [noise=true , address=104.36.82.44:4892 , piece_id=FXH2ZZHACDLSQCGJ4NBVVGLYBDDKA5J7KXT4RKUWJSBR2FOT7JUA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.123127372 (13h33m55.638229942s) [noise=true , address=104.36.82.47:4520 , piece_id=TS2GSEHDUCPNVAJJGGO4HBJQYAY4MUWN2LK3B7EFI5IJ7BHUP5HQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.125615705 (13h33m55.640718274s) [noise=true , address=104.36.82.60:4365 , piece_id=F5LBKY4MEJI5AA26CHAHYT7D47B5SJU2SVBVHUMKRPKOH2YGLFRQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.129141651 (13h33m55.644244214s) [noise=true , address=104.36.82.60:4357 , piece_id=UTRV3ECM4VKIMVI25QOD3GA7MLBO247ZDXYL7ZOPLOMM2RLSU6VQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.135781424 (13h33m55.650883986s) [noise=true , address=104.36.82.5:4844 , piece_id=REIROBOO56LH63HN6GTX237VQ4KYEACO2LXJGQ4KWPC2VQVEIPFA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145947229 (13h33m55.661049806s) [noise=true , address=104.36.82.12:4167 , piece_id=WJNPLTBMBLD7Z2ECP5SSJPDTHH7TX6U24T2OKTS7NUYIYN3C3C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145973097 (13h33m55.661075674s) [seg_pos={0 0}, upload=13963]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:21.146446342 (13h33m55.661548919s) [noise=true , address=104.36.82.39:4752 , piece_id=XXGMFHJX2IRCNBOYINHMKCJT5YS2Y72W6PL2DJQORRSHJ4OYFBCQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection Aug 7 23:13:21.146527095 (13h33m55.661629665s) [noise=true , address=79.127.205.233:4629 , piece_id=JO4HOXAE4TUR44R55JVMESPMDHZVPRZUB2UI4RSPBXZWYPWUYO6Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fCtaroi1DDvaKo4kVVMLPpwDiLWwz377LEs7pyh3BHhvkiifX): protocol: read tcp 159.223.143.39:50930->79.127.205.233:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146720823 (13h33m55.661823393s) [noise=true , address=104.36.82.9:4009 , piece_id=WSQW7NZSRPNYA4AM4T2A5T6KUWEHJJ47E6D66Y4S76RIX2KY7TMQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb): protocol: read tcp 159.223.143.39:49842->104.36.82.9:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146939426 (13h33m55.662041993s) [noise=true , address=64.49.13.154:4936 , piece_id=CP354MRPGNHBGBGMDDXCQWV4HMXLDE2BX5N3SDRI25757TH7LPJA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J4nbkWW5oM17EbRwB53vgPzEiK8dEXM2tR2TWGhJFqEja9RHi): protocol: read tcp 159.223.143.39:39690->64.49.13.154:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:13:21.146954418 (13h33m55.662056994s) [seg_pos={0 0}, upload=13963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:21.147046940 (13h33m55.662149506s) [upload=13963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:21.147051626 (13h33m55.662154193s) [upload=13963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:21.197764982 (13h33m55.712867549s) [upload=13963]: Done uploading segments... Aug 7 23:13:21.197785171 (13h33m55.712887737s) [upload=13963]: Upload finished. err: <nil> Aug 7 23:13:52.912177255 (13h34m27.427279825s) [upload=13973]: Starting upload Aug 7 23:13:52.912245188 (13h34m27.427347764s) [upload=13973]: Done starting upload Aug 7 23:13:52.912322905 (13h34m27.427425471s) [upload=13973]: Uploading segments... Aug 7 23:13:52.912741963 (13h34m27.427844541s) [upload=13973]: Got next segment. Inline: false Aug 7 23:13:52.912787686 (13h34m27.427890271s) [upload=13973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:52.912794152 (13h34m27.427896731s) [upload=13973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:52.879876232 (13h35m27.394978808s) [upload=13973]: Next segment err: context deadline exceeded Aug 7 23:14:52.879935785 (13h35m27.395038353s) [upload=13973]: Done uploading segments... Aug 7 23:14:52.879952894 (13h35m27.395055462s) [upload=13973]: Upload finished. err: context deadline exceeded Aug 7 23:14:56.273083754 (13h35m30.78818632s) [upload=13987]: Starting upload Aug 7 23:14:56.273138206 (13h35m30.788240791s) [upload=13987]: Done starting upload Aug 7 23:14:56.273487156 (13h35m30.788589726s) [upload=13987]: Uploading segments... Aug 7 23:14:56.273528218 (13h35m30.788630804s) [upload=13987]: Got next segment. Inline: false Aug 7 23:14:56.273572431 (13h35m30.788675008s) [upload=13987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.273577073 (13h35m30.788679642s) [upload=13987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:56.244079248 (13h36m30.759181825s) [upload=13987]: Next segment err: context deadline exceeded Aug 7 23:15:56.244117923 (13h36m30.7592205s) [upload=13987]: Done uploading segments... Aug 7 23:15:56.244134836 (13h36m30.759237404s) [upload=13987]: Upload finished. err: context deadline exceeded Aug 7 23:15:56.354114004 (13h36m30.869216575s) [upload=13999]: Starting upload Aug 7 23:15:56.354173464 (13h36m30.869276042s) [upload=13999]: Done starting upload Aug 7 23:15:56.354235645 (13h36m30.869338234s) [upload=13999]: Uploading segments... Aug 7 23:15:56.354444202 (13h36m30.869546778s) [upload=13999]: Got next segment. Inline: false Aug 7 23:15:56.354481122 (13h36m30.869583698s) [upload=13999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:56.354484796 (13h36m30.869587372s) [upload=13999]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:56.323532882 (13h37m30.838635479s) [upload=13999]: Next segment err: context deadline exceeded Aug 7 23:16:56.323738594 (13h37m30.838841171s) [upload=13999]: Done uploading segments... Aug 7 23:16:56.323774957 (13h37m30.838877535s) [upload=13999]: Upload finished. err: context deadline exceeded Aug 7 23:17:01.850987007 (13h37m36.366089575s) [upload=14013]: Starting upload Aug 7 23:17:01.851135604 (13h37m36.366238172s) [upload=14013]: Done starting upload Aug 7 23:17:01.851286857 (13h37m36.366389428s) [upload=14013]: Uploading segments... Aug 7 23:17:01.851780544 (13h37m36.366883126s) [upload=14013]: Got next segment. Inline: false Aug 7 23:17:01.851834378 (13h37m36.366936962s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:01.851851393 (13h37m36.366953977s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:01.818716518 (13h38m36.333819106s) [upload=14013]: Next segment err: context deadline exceeded Aug 7 23:18:01.818844791 (13h38m36.333947369s) [upload=14013]: Done uploading segments... Aug 7 23:18:01.818870437 (13h38m36.33397301s) [upload=14013]: Upload finished. err: context deadline exceeded Duration: 1m0.001263937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XX.XXX..] duration: 708.408886ms FAIL (2025-08-07 23:17:58 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30m31.135428317s) [upload=13902]: Starting upload Aug 7 23:09:56.620515140 (13h30m31.135617722s) [upload=13902]: Done starting upload Aug 7 23:09:56.620600541 (13h30m31.135703106s) [upload=13902]: Uploading segments... Aug 7 23:09:56.620663979 (13h30m31.135766559s) [upload=13902]: Got next segment. Inline: true Aug 7 23:09:56.620669313 (13h30m31.13577189s) [upload=13902]: Waiting for error group managing segments... Aug 7 23:09:56.620725144 (13h30m31.13582772s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:56.620729700 (13h30m31.135832276s) [upload=13902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:56.620732858 (13h30m31.135835435s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:56.756931332 (13h30m31.272033912s) [upload=13902]: Done uploading segments... Aug 7 23:09:56.756952203 (13h30m31.272054766s) [upload=13902]: Upload finished. err: <nil> Aug 7 23:10:22.255312070 (13h30m56.770414636s) [upload=13913]: Starting upload Aug 7 23:10:22.255389033 (13h30m56.770491609s) [upload=13913]: Done starting upload Aug 7 23:10:22.255538785 (13h30m56.77064136s) [upload=13913]: Uploading segments... Aug 7 23:10:22.255632108 (13h30m56.77073469s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:22.255639107 (13h30m56.770741683s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:22.255686584 (13h30m56.77078915s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:22.255691754 (13h30m56.770794331s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:22.255695470 (13h30m56.770798046s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:22.322045125 (13h30m56.837147702s) [upload=13913]: Done uploading segments... Aug 7 23:10:22.322068753 (13h30m56.837171329s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:00.342239204 (13h31m34.85734177s) [upload=13922]: Starting upload Aug 7 23:11:00.342310667 (13h31m34.857413235s) [upload=13922]: Done starting upload Aug 7 23:11:00.342383830 (13h31m34.85748641s) [upload=13922]: Uploading segments... Aug 7 23:11:00.342449267 (13h31m34.857551854s) [upload=13922]: Got next segment. Inline: true Aug 7 23:11:00.342455504 (13h31m34.857558092s) [upload=13922]: Waiting for error group managing segments... Aug 7 23:11:00.342503627 (13h31m34.857606209s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:00.342508465 (13h31m34.857611046s) [upload=13922]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:00.342511859 (13h31m34.857614451s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:00.401913127 (13h31m34.917015712s) [upload=13922]: Done uploading segments... Aug 7 23:11:00.401931904 (13h31m34.917034472s) [upload=13922]: Upload finished. err: <nil> Aug 7 23:11:35.523175401 (13h32m10.03827798s) [upload=13934]: Starting upload Aug 7 23:11:35.523266566 (13h32m10.038369147s) [upload=13934]: Done starting upload Aug 7 23:11:35.523355068 (13h32m10.038457647s) [upload=13934]: Uploading segments... Aug 7 23:11:35.523405664 (13h32m10.03850825s) [upload=13934]: Got next segment. Inline: true Aug 7 23:11:35.523412393 (13h32m10.038514977s) [upload=13934]: Waiting for error group managing segments... Aug 7 23:11:35.523470368 (13h32m10.038572947s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:35.523479499 (13h32m10.038582083s) [upload=13934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:35.523484240 (13h32m10.038586821s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:35.587656651 (13h32m10.102759216s) [upload=13934]: Done uploading segments... Aug 7 23:11:35.587674771 (13h32m10.102777338s) [upload=13934]: Upload finished. err: <nil> Aug 7 23:12:23.226340957 (13h32m57.741443533s) [upload=13947]: Starting upload Aug 7 23:12:23.226411041 (13h32m57.741513618s) [upload=13947]: Done starting upload Aug 7 23:12:23.226470575 (13h32m57.74157314s) [upload=13947]: Uploading segments... Aug 7 23:12:23.226506745 (13h32m57.741609326s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:23.226511437 (13h32m57.741614019s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:23.226561258 (13h32m57.741663839s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:23.226566182 (13h32m57.741668762s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:23.226569380 (13h32m57.741671956s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:53.198277272 (13h33m27.713379848s) [upload=13947]: Done uploading segments... Aug 7 23:12:53.198295252 (13h33m27.713397814s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:53.310207725 (13h33m27.825310302s) [upload=13957]: Starting upload Aug 7 23:12:53.310290257 (13h33m27.82539282s) [upload=13957]: Done starting upload Aug 7 23:12:53.310379699 (13h33m27.825482276s) [upload=13957]: Uploading segments... Aug 7 23:12:53.310488747 (13h33m27.825591336s) [upload=13957]: Got next segment. Inline: true Aug 7 23:12:53.310497967 (13h33m27.825600548s) [upload=13957]: Waiting for error group managing segments... Aug 7 23:12:53.310564975 (13h33m27.825667562s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:53.310572430 (13h33m27.825675017s) [upload=13957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:53.310576051 (13h33m27.825678628s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:23.280527765 (13h33m57.79563034s) [upload=13957]: Done uploading segments... Aug 7 23:13:23.280561185 (13h33m57.795663762s) [upload=13957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:50.810735096 (13h34m25.325837673s) [upload=13972]: Starting upload Aug 7 23:13:50.810789852 (13h34m25.325892437s) [upload=13972]: Done starting upload Aug 7 23:13:50.810850987 (13h34m25.325953556s) [upload=13972]: Uploading segments... Aug 7 23:13:50.810923216 (13h34m25.326025797s) [upload=13972]: Got next segment. Inline: true Aug 7 23:13:50.810928523 (13h34m25.326031096s) [upload=13972]: Waiting for error group managing segments... Aug 7 23:13:50.810970440 (13h34m25.326073016s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:50.810974710 (13h34m25.326077286s) [upload=13972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:50.810978274 (13h34m25.32608085s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:50.866577223 (13h34m25.3816798s) [upload=13972]: Done uploading segments... Aug 7 23:13:50.866594654 (13h34m25.38169723s) [upload=13972]: Upload finished. err: <nil> Aug 7 23:14:54.225692687 (13h35m28.740795254s) [upload=13986]: Starting upload Aug 7 23:14:54.225785134 (13h35m28.740887721s) [upload=13986]: Done starting upload Aug 7 23:14:54.225870963 (13h35m28.740973529s) [upload=13986]: Uploading segments... Aug 7 23:14:54.225923809 (13h35m28.741026386s) [upload=13986]: Got next segment. Inline: true Aug 7 23:14:54.225928631 (13h35m28.741031207s) [upload=13986]: Waiting for error group managing segments... Aug 7 23:14:54.225969273 (13h35m28.741071842s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:54.225973734 (13h35m28.7410763s) [upload=13986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:54.225976991 (13h35m28.741079566s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:24.191802298 (13h35m58.706904867s) [upload=13986]: Done uploading segments... Aug 7 23:15:24.191822370 (13h35m58.70692494s) [upload=13986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:24.446619359 (13h35m58.961721924s) [upload=13992]: Starting upload Aug 7 23:15:24.446725754 (13h35m58.961828338s) [upload=13992]: Done starting upload Aug 7 23:15:24.446812887 (13h35m58.961915463s) [upload=13992]: Uploading segments... Aug 7 23:15:24.446871932 (13h35m58.961974519s) [upload=13992]: Got next segment. Inline: true Aug 7 23:15:24.446878746 (13h35m58.961981315s) [upload=13992]: Waiting for error group managing segments... Aug 7 23:15:24.446939653 (13h35m58.962042222s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:24.446945288 (13h35m58.962047863s) [upload=13992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:24.446950467 (13h35m58.96205303s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:54.414392139 (13h36m28.929494705s) [upload=13992]: Done uploading segments... Aug 7 23:15:54.414412085 (13h36m28.929514651s) [upload=13992]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:32.150263694 (13h37m6.665366261s) [upload=14009]: Starting upload Aug 7 23:16:32.150345034 (13h37m6.665447612s) [upload=14009]: Done starting upload Aug 7 23:16:32.150402507 (13h37m6.665505071s) [upload=14009]: Uploading segments... Aug 7 23:16:32.150453597 (13h37m6.665556176s) [upload=14009]: Got next segment. Inline: true Aug 7 23:16:32.150458385 (13h37m6.665560958s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:16:32.150501065 (13h37m6.665603641s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:32.150505833 (13h37m6.66560841s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:32.150509597 (13h37m6.665612174s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:02.044102103 (13h37m36.55920467s) [upload=14009]: Done uploading segments... Aug 7 23:17:02.044123177 (13h37m36.559225743s) [upload=14009]: Upload finished. err: metaclient: context deadline exceeded Duration: 708.429531ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XX.X.X.] duration: 611.599684ms FAIL (2025-08-07 23:17:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 611.627041ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 2.154806205s FAIL (2025-08-07 23:17:48 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.154868184s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 180.050771ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:17:44 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 180.0659ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................X.XXX.X] duration: 1m0.00167147s FAIL (2025-08-07 23:17:44 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001693687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................................................X..XXXXX] duration: 30.001445675s FAIL (2025-08-07 23:17:33 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001467568s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 2m0.000747702s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:17:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.00077088s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 1.975978145s FAIL (2025-08-07 23:17:19 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.976022327s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.333571514s FAIL (2025-08-07 23:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== g_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214654258 (12h54m14.729756841s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214886232 (12h54m14.729988875s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215184221 (12h54m14.730286807s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215580311 (12h54m14.730682897s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215600089 (12h54m14.730702668s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216063638 (12h54m14.731166221s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216221465 (12h54m14.73132405s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216499611 (12h54m14.73160219s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216784173 (12h54m14.731886753s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216959396 (12h54m14.732061963s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217229150 (12h54m14.732331732s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217364444 (12h54m14.732467028s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217574188 (12h54m14.732676766s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.218700401 (12h54m14.733802981s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.219113148 (12h54m14.734215718s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.307711467 (12h54m14.82281403s) [upload=13273]: Next returned nil segment Aug 7 22:33:40.307743585 (12h54m14.822846164s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:33:40.321114866 (12h54m14.836217441s) [noise=true , address=45.91.116.9:4294 , piece_id=UMCVXJVJ7VWVLOXYVN4KSE5VHM2YXJWOD45364KGN2T62PGBYNOA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.321899105 (12h54m14.837001676s) [noise=true , address=45.91.116.4:4943 , piece_id=4U7YPCLZI7HIZGGBGVWIWJLBRNPGIYOVVUD2UEJJVCMZMC426JIA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.330294085 (12h54m14.845396652s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.331982437 (12h54m14.847085014s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.334537739 (12h54m14.849640313s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.335955579 (12h54m14.851058147s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.340231956 (12h54m14.855334526s) [noise=true , address=77.90.31.11:4559 , piece_id=JSL5C3XIXINYAVAVTQ7J3QD3WWRG6FKP4H4DGJLDOPWCGI7CNAFA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.349513208 (12h54m14.864615777s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.358627538 (12h54m14.873730105s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.436339846 (12h54m14.951442414s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.543248537 (12h54m15.058351106s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.650678621 (12h54m15.165781209s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.786445525 (12h54m15.301548105s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907165087 (12h54m15.422267656s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907202527 (12h54m15.422305104s) [seg_pos={0 0}, upload=13273]: Segment reached optimal threshold of 14 pieces. Aug 7 22:33:40.907372225 (12h54m15.422474801s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QrrGrMHYAjyYiTdTXDUy2uH1j2p1qtUVJaDtais2GowJqBPPu): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.907642442 (12h54m15.422745018s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T): protocol: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection Aug 7 22:33:40.907950578 (12h54m15.423053162s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection Aug 7 22:33:40.908057652 (12h54m15.423160235s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.908080526 (12h54m15.423183103s) [seg_pos={0 0}, upload=13273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:33:40.908212573 (12h54m15.423315155s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:33:40.908218832 (12h54m15.423321415s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:41.213299020 (12h54m15.728401607s) [upload=13273]: Done uploading segments... Aug 7 22:33:41.213326550 (12h54m15.728429128s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:34:48.031458911 (12h55m22.546561477s) [upload=13295]: Starting upload Aug 7 22:34:48.031533867 (12h55m22.546636441s) [upload=13295]: Done starting upload Aug 7 22:34:48.031664339 (12h55m22.546766908s) [upload=13295]: Uploading segments... Aug 7 22:34:48.031895444 (12h55m22.546998028s) [upload=13295]: Got next segment. Inline: false Aug 7 22:34:48.031982233 (12h55m22.547084799s) [upload=13295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:48.031988106 (12h55m22.547090688s) [upload=13295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:34:48.040357077 (12h55m22.555459659s) [upload=13295]: Next segment err: metaclient: Too Many Requests Aug 7 22:34:48.040386485 (12h55m22.555489061s) [upload=13295]: Done uploading segments... Aug 7 22:34:48.040426787 (12h55m22.555529363s) [upload=13295]: Upload finished. err: metaclient: Too Many Requests Duration: 1.333604375s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XX..X.] duration: 242.195533ms FAIL (2025-08-07 23:17:16 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad finished. err: <nil> Aug 7 23:07:56.297738414 (13h28m30.812840994s) [upload=13869]: Starting upload Aug 7 23:07:56.297858483 (13h28m30.812961062s) [upload=13869]: Done starting upload Aug 7 23:07:56.297993444 (13h28m30.813096021s) [upload=13869]: Uploading segments... Aug 7 23:07:56.298078193 (13h28m30.813180778s) [upload=13869]: Got next segment. Inline: true Aug 7 23:07:56.298102093 (13h28m30.813204678s) [upload=13869]: Waiting for error group managing segments... Aug 7 23:07:56.298156325 (13h28m30.813258907s) [upload=13869]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:56.298177912 (13h28m30.813280494s) [upload=13869]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:56.298182546 (13h28m30.813285127s) [upload=13869]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:56.372043207 (13h28m30.887145784s) [upload=13869]: Done uploading segments... Aug 7 23:07:56.372061460 (13h28m30.887164036s) [upload=13869]: Upload finished. err: <nil> Aug 7 23:08:37.049508399 (13h29m11.564610963s) [upload=13879]: Starting upload Aug 7 23:08:37.049588017 (13h29m11.564690596s) [upload=13879]: Done starting upload Aug 7 23:08:37.049683838 (13h29m11.564786416s) [upload=13879]: Uploading segments... Aug 7 23:08:37.049829834 (13h29m11.564932424s) [upload=13879]: Got next segment. Inline: true Aug 7 23:08:37.049839426 (13h29m11.564942003s) [upload=13879]: Waiting for error group managing segments... Aug 7 23:08:37.049910072 (13h29m11.565012644s) [upload=13879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:37.049916484 (13h29m11.56501906s) [upload=13879]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:37.049920022 (13h29m11.565022598s) [upload=13879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:37.140370137 (13h29m11.655472717s) [upload=13879]: Done uploading segments... Aug 7 23:08:37.141250272 (13h29m11.65635286s) [upload=13879]: Upload finished. err: <nil> Aug 7 23:09:29.216369700 (13h30m3.731472269s) [upload=13895]: Starting upload Aug 7 23:09:29.216489949 (13h30m3.731592525s) [upload=13895]: Done starting upload Aug 7 23:09:29.216567331 (13h30m3.731669896s) [upload=13895]: Uploading segments... Aug 7 23:09:29.216612331 (13h30m3.731714917s) [upload=13895]: Got next segment. Inline: true Aug 7 23:09:29.216618125 (13h30m3.731720692s) [upload=13895]: Waiting for error group managing segments... Aug 7 23:09:29.216659267 (13h30m3.731761843s) [upload=13895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:29.216663938 (13h30m3.731766517s) [upload=13895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:29.216668553 (13h30m3.731771139s) [upload=13895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:29.297928170 (13h30m3.813030739s) [upload=13895]: Done uploading segments... Aug 7 23:09:29.297957487 (13h30m3.813060063s) [upload=13895]: Upload finished. err: <nil> Aug 7 23:10:23.115383547 (13h30m57.630486132s) [upload=13915]: Starting upload Aug 7 23:10:23.115456203 (13h30m57.630558786s) [upload=13915]: Done starting upload Aug 7 23:10:23.115557871 (13h30m57.630660452s) [upload=13915]: Uploading segments... Aug 7 23:10:23.115614526 (13h30m57.630717116s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:23.115621076 (13h30m57.630723665s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:23.115685040 (13h30m57.630787627s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:23.115692213 (13h30m57.630794802s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:23.115697119 (13h30m57.630799709s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:23.183146377 (13h30m57.698248962s) [upload=13915]: Done uploading segments... Aug 7 23:10:23.183169407 (13h30m57.698271988s) [upload=13915]: Upload finished. err: <nil> Aug 7 23:10:46.162725354 (13h31m20.67782793s) [upload=13919]: Starting upload Aug 7 23:10:46.162807728 (13h31m20.677910304s) [upload=13919]: Done starting upload Aug 7 23:10:46.162877771 (13h31m20.677980347s) [upload=13919]: Uploading segments... Aug 7 23:10:46.162939319 (13h31m20.678041901s) [upload=13919]: Got next segment. Inline: true Aug 7 23:10:46.162944044 (13h31m20.678046624s) [upload=13919]: Waiting for error group managing segments... Aug 7 23:10:46.162986704 (13h31m20.678089286s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:46.162991049 (13h31m20.678093629s) [upload=13919]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:46.162994213 (13h31m20.678096792s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:49.331481110 (13h31m23.84658369s) [upload=13919]: Done uploading segments... Aug 7 23:10:49.331502710 (13h31m23.846605285s) [upload=13919]: Upload finished. err: <nil> Aug 7 23:11:56.588799333 (13h32m31.103901909s) [upload=13942]: Starting upload Aug 7 23:11:56.588871172 (13h32m31.103973747s) [upload=13942]: Done starting upload Aug 7 23:11:56.588959193 (13h32m31.104061755s) [upload=13942]: Uploading segments... Aug 7 23:11:56.588996486 (13h32m31.104099072s) [upload=13942]: Got next segment. Inline: true Aug 7 23:11:56.589000914 (13h32m31.104103491s) [upload=13942]: Waiting for error group managing segments... Aug 7 23:11:56.589047870 (13h32m31.104150433s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.589051856 (13h32m31.104154432s) [upload=13942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.589054372 (13h32m31.10415695s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:56.555383755 (13h33m31.070486344s) [upload=13942]: Done uploading segments... Aug 7 23:12:56.555407795 (13h33m31.070510378s) [upload=13942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:56.669023639 (13h33m31.184126201s) [upload=13959]: Starting upload Aug 7 23:12:56.669081512 (13h33m31.184184088s) [upload=13959]: Done starting upload Aug 7 23:12:56.669139265 (13h33m31.184241827s) [upload=13959]: Uploading segments... Aug 7 23:12:56.669181725 (13h33m31.184284307s) [upload=13959]: Got next segment. Inline: true Aug 7 23:12:56.669185727 (13h33m31.18428829s) [upload=13959]: Waiting for error group managing segments... Aug 7 23:12:56.669222629 (13h33m31.184325191s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:56.669226085 (13h33m31.18432866s) [upload=13959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:56.669228556 (13h33m31.184331118s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.639927224 (13h34m31.155029791s) [upload=13959]: Done uploading segments... Aug 7 23:13:56.639946863 (13h34m31.155049429s) [upload=13959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:56.754611797 (13h34m31.269714367s) [upload=13974]: Starting upload Aug 7 23:13:56.754690544 (13h34m31.269793123s) [upload=13974]: Done starting upload Aug 7 23:13:56.754787910 (13h34m31.269890478s) [upload=13974]: Uploading segments... Aug 7 23:13:56.754829279 (13h34m31.269931864s) [upload=13974]: Got next segment. Inline: true Aug 7 23:13:56.754833834 (13h34m31.269936411s) [upload=13974]: Waiting for error group managing segments... Aug 7 23:13:56.754887650 (13h34m31.269990217s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.754891993 (13h34m31.26999456s) [upload=13974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.754895495 (13h34m31.269998071s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.814228870 (13h34m31.32933145s) [upload=13974]: Done uploading segments... Aug 7 23:13:56.814253732 (13h34m31.329356295s) [upload=13974]: Upload finished. err: <nil> Aug 7 23:14:51.237083244 (13h35m25.75218581s) [upload=13984]: Starting upload Aug 7 23:14:51.237154440 (13h35m25.752257016s) [upload=13984]: Done starting upload Aug 7 23:14:51.237222602 (13h35m25.752325168s) [upload=13984]: Uploading segments... Aug 7 23:14:51.237272573 (13h35m25.752375158s) [upload=13984]: Got next segment. Inline: true Aug 7 23:14:51.237277088 (13h35m25.752379654s) [upload=13984]: Waiting for error group managing segments... Aug 7 23:14:51.237328372 (13h35m25.752430937s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:51.237332974 (13h35m25.75243555s) [upload=13984]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:51.237336117 (13h35m25.752438684s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:51.367347705 (13h35m25.882450281s) [upload=13984]: Done uploading segments... Aug 7 23:14:51.367368302 (13h35m25.882470865s) [upload=13984]: Upload finished. err: <nil> Aug 7 23:16:05.237199068 (13h36m39.752301655s) [upload=14002]: Starting upload Aug 7 23:16:05.237269888 (13h36m39.752372457s) [upload=14002]: Done starting upload Aug 7 23:16:05.237360382 (13h36m39.752462952s) [upload=14002]: Uploading segments... Aug 7 23:16:05.237841750 (13h36m39.752944336s) [upload=14002]: Got next segment. Inline: true Aug 7 23:16:05.237859632 (13h36m39.752962218s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:16:05.237930004 (13h36m39.753032586s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.237936533 (13h36m39.753039109s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.237940573 (13h36m39.753043148s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.205359206 (13h37m39.720461782s) [upload=14002]: Done uploading segments... Aug 7 23:17:05.205376470 (13h37m39.720479046s) [upload=14002]: Upload finished. err: metaclient: context deadline exceeded Duration: 242.220553ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XX.X.X] duration: 30.001170948s FAIL (2025-08-07 23:17:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001190772s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XX.XXX.] duration: 239.824135ms FAIL (2025-08-07 23:17:09 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30m31.135428317s) [upload=13902]: Starting upload Aug 7 23:09:56.620515140 (13h30m31.135617722s) [upload=13902]: Done starting upload Aug 7 23:09:56.620600541 (13h30m31.135703106s) [upload=13902]: Uploading segments... Aug 7 23:09:56.620663979 (13h30m31.135766559s) [upload=13902]: Got next segment. Inline: true Aug 7 23:09:56.620669313 (13h30m31.13577189s) [upload=13902]: Waiting for error group managing segments... Aug 7 23:09:56.620725144 (13h30m31.13582772s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:56.620729700 (13h30m31.135832276s) [upload=13902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:56.620732858 (13h30m31.135835435s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:56.756931332 (13h30m31.272033912s) [upload=13902]: Done uploading segments... Aug 7 23:09:56.756952203 (13h30m31.272054766s) [upload=13902]: Upload finished. err: <nil> Aug 7 23:10:22.255312070 (13h30m56.770414636s) [upload=13913]: Starting upload Aug 7 23:10:22.255389033 (13h30m56.770491609s) [upload=13913]: Done starting upload Aug 7 23:10:22.255538785 (13h30m56.77064136s) [upload=13913]: Uploading segments... Aug 7 23:10:22.255632108 (13h30m56.77073469s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:22.255639107 (13h30m56.770741683s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:22.255686584 (13h30m56.77078915s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:22.255691754 (13h30m56.770794331s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:22.255695470 (13h30m56.770798046s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:22.322045125 (13h30m56.837147702s) [upload=13913]: Done uploading segments... Aug 7 23:10:22.322068753 (13h30m56.837171329s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:00.342239204 (13h31m34.85734177s) [upload=13922]: Starting upload Aug 7 23:11:00.342310667 (13h31m34.857413235s) [upload=13922]: Done starting upload Aug 7 23:11:00.342383830 (13h31m34.85748641s) [upload=13922]: Uploading segments... Aug 7 23:11:00.342449267 (13h31m34.857551854s) [upload=13922]: Got next segment. Inline: true Aug 7 23:11:00.342455504 (13h31m34.857558092s) [upload=13922]: Waiting for error group managing segments... Aug 7 23:11:00.342503627 (13h31m34.857606209s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:00.342508465 (13h31m34.857611046s) [upload=13922]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:00.342511859 (13h31m34.857614451s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:00.401913127 (13h31m34.917015712s) [upload=13922]: Done uploading segments... Aug 7 23:11:00.401931904 (13h31m34.917034472s) [upload=13922]: Upload finished. err: <nil> Aug 7 23:11:35.523175401 (13h32m10.03827798s) [upload=13934]: Starting upload Aug 7 23:11:35.523266566 (13h32m10.038369147s) [upload=13934]: Done starting upload Aug 7 23:11:35.523355068 (13h32m10.038457647s) [upload=13934]: Uploading segments... Aug 7 23:11:35.523405664 (13h32m10.03850825s) [upload=13934]: Got next segment. Inline: true Aug 7 23:11:35.523412393 (13h32m10.038514977s) [upload=13934]: Waiting for error group managing segments... Aug 7 23:11:35.523470368 (13h32m10.038572947s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:35.523479499 (13h32m10.038582083s) [upload=13934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:35.523484240 (13h32m10.038586821s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:35.587656651 (13h32m10.102759216s) [upload=13934]: Done uploading segments... Aug 7 23:11:35.587674771 (13h32m10.102777338s) [upload=13934]: Upload finished. err: <nil> Aug 7 23:12:23.226340957 (13h32m57.741443533s) [upload=13947]: Starting upload Aug 7 23:12:23.226411041 (13h32m57.741513618s) [upload=13947]: Done starting upload Aug 7 23:12:23.226470575 (13h32m57.74157314s) [upload=13947]: Uploading segments... Aug 7 23:12:23.226506745 (13h32m57.741609326s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:23.226511437 (13h32m57.741614019s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:23.226561258 (13h32m57.741663839s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:23.226566182 (13h32m57.741668762s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:23.226569380 (13h32m57.741671956s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:53.198277272 (13h33m27.713379848s) [upload=13947]: Done uploading segments... Aug 7 23:12:53.198295252 (13h33m27.713397814s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:53.310207725 (13h33m27.825310302s) [upload=13957]: Starting upload Aug 7 23:12:53.310290257 (13h33m27.82539282s) [upload=13957]: Done starting upload Aug 7 23:12:53.310379699 (13h33m27.825482276s) [upload=13957]: Uploading segments... Aug 7 23:12:53.310488747 (13h33m27.825591336s) [upload=13957]: Got next segment. Inline: true Aug 7 23:12:53.310497967 (13h33m27.825600548s) [upload=13957]: Waiting for error group managing segments... Aug 7 23:12:53.310564975 (13h33m27.825667562s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:53.310572430 (13h33m27.825675017s) [upload=13957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:53.310576051 (13h33m27.825678628s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:23.280527765 (13h33m57.79563034s) [upload=13957]: Done uploading segments... Aug 7 23:13:23.280561185 (13h33m57.795663762s) [upload=13957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:50.810735096 (13h34m25.325837673s) [upload=13972]: Starting upload Aug 7 23:13:50.810789852 (13h34m25.325892437s) [upload=13972]: Done starting upload Aug 7 23:13:50.810850987 (13h34m25.325953556s) [upload=13972]: Uploading segments... Aug 7 23:13:50.810923216 (13h34m25.326025797s) [upload=13972]: Got next segment. Inline: true Aug 7 23:13:50.810928523 (13h34m25.326031096s) [upload=13972]: Waiting for error group managing segments... Aug 7 23:13:50.810970440 (13h34m25.326073016s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:50.810974710 (13h34m25.326077286s) [upload=13972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:50.810978274 (13h34m25.32608085s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:50.866577223 (13h34m25.3816798s) [upload=13972]: Done uploading segments... Aug 7 23:13:50.866594654 (13h34m25.38169723s) [upload=13972]: Upload finished. err: <nil> Aug 7 23:14:54.225692687 (13h35m28.740795254s) [upload=13986]: Starting upload Aug 7 23:14:54.225785134 (13h35m28.740887721s) [upload=13986]: Done starting upload Aug 7 23:14:54.225870963 (13h35m28.740973529s) [upload=13986]: Uploading segments... Aug 7 23:14:54.225923809 (13h35m28.741026386s) [upload=13986]: Got next segment. Inline: true Aug 7 23:14:54.225928631 (13h35m28.741031207s) [upload=13986]: Waiting for error group managing segments... Aug 7 23:14:54.225969273 (13h35m28.741071842s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:54.225973734 (13h35m28.7410763s) [upload=13986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:54.225976991 (13h35m28.741079566s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:24.191802298 (13h35m58.706904867s) [upload=13986]: Done uploading segments... Aug 7 23:15:24.191822370 (13h35m58.70692494s) [upload=13986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:24.446619359 (13h35m58.961721924s) [upload=13992]: Starting upload Aug 7 23:15:24.446725754 (13h35m58.961828338s) [upload=13992]: Done starting upload Aug 7 23:15:24.446812887 (13h35m58.961915463s) [upload=13992]: Uploading segments... Aug 7 23:15:24.446871932 (13h35m58.961974519s) [upload=13992]: Got next segment. Inline: true Aug 7 23:15:24.446878746 (13h35m58.961981315s) [upload=13992]: Waiting for error group managing segments... Aug 7 23:15:24.446939653 (13h35m58.962042222s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:24.446945288 (13h35m58.962047863s) [upload=13992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:24.446950467 (13h35m58.96205303s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:54.414392139 (13h36m28.929494705s) [upload=13992]: Done uploading segments... Aug 7 23:15:54.414412085 (13h36m28.929514651s) [upload=13992]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:32.150263694 (13h37m6.665366261s) [upload=14009]: Starting upload Aug 7 23:16:32.150345034 (13h37m6.665447612s) [upload=14009]: Done starting upload Aug 7 23:16:32.150402507 (13h37m6.665505071s) [upload=14009]: Uploading segments... Aug 7 23:16:32.150453597 (13h37m6.665556176s) [upload=14009]: Got next segment. Inline: true Aug 7 23:16:32.150458385 (13h37m6.665560958s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:16:32.150501065 (13h37m6.665603641s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:32.150505833 (13h37m6.66560841s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:32.150509597 (13h37m6.665612174s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:02.044102103 (13h37m36.55920467s) [upload=14009]: Done uploading segments... Aug 7 23:17:02.044123177 (13h37m36.559225743s) [upload=14009]: Upload finished. err: metaclient: context deadline exceeded Duration: 239.917627ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX..X] duration: 1m0.001222072s FAIL (2025-08-07 23:17:05 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad finished. err: <nil> Aug 7 23:07:56.297738414 (13h28m30.812840994s) [upload=13869]: Starting upload Aug 7 23:07:56.297858483 (13h28m30.812961062s) [upload=13869]: Done starting upload Aug 7 23:07:56.297993444 (13h28m30.813096021s) [upload=13869]: Uploading segments... Aug 7 23:07:56.298078193 (13h28m30.813180778s) [upload=13869]: Got next segment. Inline: true Aug 7 23:07:56.298102093 (13h28m30.813204678s) [upload=13869]: Waiting for error group managing segments... Aug 7 23:07:56.298156325 (13h28m30.813258907s) [upload=13869]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:56.298177912 (13h28m30.813280494s) [upload=13869]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:56.298182546 (13h28m30.813285127s) [upload=13869]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:56.372043207 (13h28m30.887145784s) [upload=13869]: Done uploading segments... Aug 7 23:07:56.372061460 (13h28m30.887164036s) [upload=13869]: Upload finished. err: <nil> Aug 7 23:08:37.049508399 (13h29m11.564610963s) [upload=13879]: Starting upload Aug 7 23:08:37.049588017 (13h29m11.564690596s) [upload=13879]: Done starting upload Aug 7 23:08:37.049683838 (13h29m11.564786416s) [upload=13879]: Uploading segments... Aug 7 23:08:37.049829834 (13h29m11.564932424s) [upload=13879]: Got next segment. Inline: true Aug 7 23:08:37.049839426 (13h29m11.564942003s) [upload=13879]: Waiting for error group managing segments... Aug 7 23:08:37.049910072 (13h29m11.565012644s) [upload=13879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:37.049916484 (13h29m11.56501906s) [upload=13879]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:37.049920022 (13h29m11.565022598s) [upload=13879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:37.140370137 (13h29m11.655472717s) [upload=13879]: Done uploading segments... Aug 7 23:08:37.141250272 (13h29m11.65635286s) [upload=13879]: Upload finished. err: <nil> Aug 7 23:09:29.216369700 (13h30m3.731472269s) [upload=13895]: Starting upload Aug 7 23:09:29.216489949 (13h30m3.731592525s) [upload=13895]: Done starting upload Aug 7 23:09:29.216567331 (13h30m3.731669896s) [upload=13895]: Uploading segments... Aug 7 23:09:29.216612331 (13h30m3.731714917s) [upload=13895]: Got next segment. Inline: true Aug 7 23:09:29.216618125 (13h30m3.731720692s) [upload=13895]: Waiting for error group managing segments... Aug 7 23:09:29.216659267 (13h30m3.731761843s) [upload=13895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:29.216663938 (13h30m3.731766517s) [upload=13895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:29.216668553 (13h30m3.731771139s) [upload=13895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:29.297928170 (13h30m3.813030739s) [upload=13895]: Done uploading segments... Aug 7 23:09:29.297957487 (13h30m3.813060063s) [upload=13895]: Upload finished. err: <nil> Aug 7 23:10:23.115383547 (13h30m57.630486132s) [upload=13915]: Starting upload Aug 7 23:10:23.115456203 (13h30m57.630558786s) [upload=13915]: Done starting upload Aug 7 23:10:23.115557871 (13h30m57.630660452s) [upload=13915]: Uploading segments... Aug 7 23:10:23.115614526 (13h30m57.630717116s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:23.115621076 (13h30m57.630723665s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:23.115685040 (13h30m57.630787627s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:23.115692213 (13h30m57.630794802s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:23.115697119 (13h30m57.630799709s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:23.183146377 (13h30m57.698248962s) [upload=13915]: Done uploading segments... Aug 7 23:10:23.183169407 (13h30m57.698271988s) [upload=13915]: Upload finished. err: <nil> Aug 7 23:10:46.162725354 (13h31m20.67782793s) [upload=13919]: Starting upload Aug 7 23:10:46.162807728 (13h31m20.677910304s) [upload=13919]: Done starting upload Aug 7 23:10:46.162877771 (13h31m20.677980347s) [upload=13919]: Uploading segments... Aug 7 23:10:46.162939319 (13h31m20.678041901s) [upload=13919]: Got next segment. Inline: true Aug 7 23:10:46.162944044 (13h31m20.678046624s) [upload=13919]: Waiting for error group managing segments... Aug 7 23:10:46.162986704 (13h31m20.678089286s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:46.162991049 (13h31m20.678093629s) [upload=13919]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:46.162994213 (13h31m20.678096792s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:49.331481110 (13h31m23.84658369s) [upload=13919]: Done uploading segments... Aug 7 23:10:49.331502710 (13h31m23.846605285s) [upload=13919]: Upload finished. err: <nil> Aug 7 23:11:56.588799333 (13h32m31.103901909s) [upload=13942]: Starting upload Aug 7 23:11:56.588871172 (13h32m31.103973747s) [upload=13942]: Done starting upload Aug 7 23:11:56.588959193 (13h32m31.104061755s) [upload=13942]: Uploading segments... Aug 7 23:11:56.588996486 (13h32m31.104099072s) [upload=13942]: Got next segment. Inline: true Aug 7 23:11:56.589000914 (13h32m31.104103491s) [upload=13942]: Waiting for error group managing segments... Aug 7 23:11:56.589047870 (13h32m31.104150433s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.589051856 (13h32m31.104154432s) [upload=13942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.589054372 (13h32m31.10415695s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:56.555383755 (13h33m31.070486344s) [upload=13942]: Done uploading segments... Aug 7 23:12:56.555407795 (13h33m31.070510378s) [upload=13942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:56.669023639 (13h33m31.184126201s) [upload=13959]: Starting upload Aug 7 23:12:56.669081512 (13h33m31.184184088s) [upload=13959]: Done starting upload Aug 7 23:12:56.669139265 (13h33m31.184241827s) [upload=13959]: Uploading segments... Aug 7 23:12:56.669181725 (13h33m31.184284307s) [upload=13959]: Got next segment. Inline: true Aug 7 23:12:56.669185727 (13h33m31.18428829s) [upload=13959]: Waiting for error group managing segments... Aug 7 23:12:56.669222629 (13h33m31.184325191s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:56.669226085 (13h33m31.18432866s) [upload=13959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:56.669228556 (13h33m31.184331118s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.639927224 (13h34m31.155029791s) [upload=13959]: Done uploading segments... Aug 7 23:13:56.639946863 (13h34m31.155049429s) [upload=13959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:56.754611797 (13h34m31.269714367s) [upload=13974]: Starting upload Aug 7 23:13:56.754690544 (13h34m31.269793123s) [upload=13974]: Done starting upload Aug 7 23:13:56.754787910 (13h34m31.269890478s) [upload=13974]: Uploading segments... Aug 7 23:13:56.754829279 (13h34m31.269931864s) [upload=13974]: Got next segment. Inline: true Aug 7 23:13:56.754833834 (13h34m31.269936411s) [upload=13974]: Waiting for error group managing segments... Aug 7 23:13:56.754887650 (13h34m31.269990217s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.754891993 (13h34m31.26999456s) [upload=13974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.754895495 (13h34m31.269998071s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.814228870 (13h34m31.32933145s) [upload=13974]: Done uploading segments... Aug 7 23:13:56.814253732 (13h34m31.329356295s) [upload=13974]: Upload finished. err: <nil> Aug 7 23:14:51.237083244 (13h35m25.75218581s) [upload=13984]: Starting upload Aug 7 23:14:51.237154440 (13h35m25.752257016s) [upload=13984]: Done starting upload Aug 7 23:14:51.237222602 (13h35m25.752325168s) [upload=13984]: Uploading segments... Aug 7 23:14:51.237272573 (13h35m25.752375158s) [upload=13984]: Got next segment. Inline: true Aug 7 23:14:51.237277088 (13h35m25.752379654s) [upload=13984]: Waiting for error group managing segments... Aug 7 23:14:51.237328372 (13h35m25.752430937s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:51.237332974 (13h35m25.75243555s) [upload=13984]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:51.237336117 (13h35m25.752438684s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:51.367347705 (13h35m25.882450281s) [upload=13984]: Done uploading segments... Aug 7 23:14:51.367368302 (13h35m25.882470865s) [upload=13984]: Upload finished. err: <nil> Aug 7 23:16:05.237199068 (13h36m39.752301655s) [upload=14002]: Starting upload Aug 7 23:16:05.237269888 (13h36m39.752372457s) [upload=14002]: Done starting upload Aug 7 23:16:05.237360382 (13h36m39.752462952s) [upload=14002]: Uploading segments... Aug 7 23:16:05.237841750 (13h36m39.752944336s) [upload=14002]: Got next segment. Inline: true Aug 7 23:16:05.237859632 (13h36m39.752962218s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:16:05.237930004 (13h36m39.753032586s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.237936533 (13h36m39.753039109s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.237940573 (13h36m39.753043148s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.205359206 (13h37m39.720461782s) [upload=14002]: Done uploading segments... Aug 7 23:17:05.205376470 (13h37m39.720479046s) [upload=14002]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001245672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XX.XXX] duration: 30.000976785s FAIL (2025-08-07 23:17:02 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30m31.135428317s) [upload=13902]: Starting upload Aug 7 23:09:56.620515140 (13h30m31.135617722s) [upload=13902]: Done starting upload Aug 7 23:09:56.620600541 (13h30m31.135703106s) [upload=13902]: Uploading segments... Aug 7 23:09:56.620663979 (13h30m31.135766559s) [upload=13902]: Got next segment. Inline: true Aug 7 23:09:56.620669313 (13h30m31.13577189s) [upload=13902]: Waiting for error group managing segments... Aug 7 23:09:56.620725144 (13h30m31.13582772s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:56.620729700 (13h30m31.135832276s) [upload=13902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:56.620732858 (13h30m31.135835435s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:56.756931332 (13h30m31.272033912s) [upload=13902]: Done uploading segments... Aug 7 23:09:56.756952203 (13h30m31.272054766s) [upload=13902]: Upload finished. err: <nil> Aug 7 23:10:22.255312070 (13h30m56.770414636s) [upload=13913]: Starting upload Aug 7 23:10:22.255389033 (13h30m56.770491609s) [upload=13913]: Done starting upload Aug 7 23:10:22.255538785 (13h30m56.77064136s) [upload=13913]: Uploading segments... Aug 7 23:10:22.255632108 (13h30m56.77073469s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:22.255639107 (13h30m56.770741683s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:22.255686584 (13h30m56.77078915s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:22.255691754 (13h30m56.770794331s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:22.255695470 (13h30m56.770798046s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:22.322045125 (13h30m56.837147702s) [upload=13913]: Done uploading segments... Aug 7 23:10:22.322068753 (13h30m56.837171329s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:00.342239204 (13h31m34.85734177s) [upload=13922]: Starting upload Aug 7 23:11:00.342310667 (13h31m34.857413235s) [upload=13922]: Done starting upload Aug 7 23:11:00.342383830 (13h31m34.85748641s) [upload=13922]: Uploading segments... Aug 7 23:11:00.342449267 (13h31m34.857551854s) [upload=13922]: Got next segment. Inline: true Aug 7 23:11:00.342455504 (13h31m34.857558092s) [upload=13922]: Waiting for error group managing segments... Aug 7 23:11:00.342503627 (13h31m34.857606209s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:00.342508465 (13h31m34.857611046s) [upload=13922]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:00.342511859 (13h31m34.857614451s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:00.401913127 (13h31m34.917015712s) [upload=13922]: Done uploading segments... Aug 7 23:11:00.401931904 (13h31m34.917034472s) [upload=13922]: Upload finished. err: <nil> Aug 7 23:11:35.523175401 (13h32m10.03827798s) [upload=13934]: Starting upload Aug 7 23:11:35.523266566 (13h32m10.038369147s) [upload=13934]: Done starting upload Aug 7 23:11:35.523355068 (13h32m10.038457647s) [upload=13934]: Uploading segments... Aug 7 23:11:35.523405664 (13h32m10.03850825s) [upload=13934]: Got next segment. Inline: true Aug 7 23:11:35.523412393 (13h32m10.038514977s) [upload=13934]: Waiting for error group managing segments... Aug 7 23:11:35.523470368 (13h32m10.038572947s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:35.523479499 (13h32m10.038582083s) [upload=13934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:35.523484240 (13h32m10.038586821s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:35.587656651 (13h32m10.102759216s) [upload=13934]: Done uploading segments... Aug 7 23:11:35.587674771 (13h32m10.102777338s) [upload=13934]: Upload finished. err: <nil> Aug 7 23:12:23.226340957 (13h32m57.741443533s) [upload=13947]: Starting upload Aug 7 23:12:23.226411041 (13h32m57.741513618s) [upload=13947]: Done starting upload Aug 7 23:12:23.226470575 (13h32m57.74157314s) [upload=13947]: Uploading segments... Aug 7 23:12:23.226506745 (13h32m57.741609326s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:23.226511437 (13h32m57.741614019s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:23.226561258 (13h32m57.741663839s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:23.226566182 (13h32m57.741668762s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:23.226569380 (13h32m57.741671956s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:53.198277272 (13h33m27.713379848s) [upload=13947]: Done uploading segments... Aug 7 23:12:53.198295252 (13h33m27.713397814s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:53.310207725 (13h33m27.825310302s) [upload=13957]: Starting upload Aug 7 23:12:53.310290257 (13h33m27.82539282s) [upload=13957]: Done starting upload Aug 7 23:12:53.310379699 (13h33m27.825482276s) [upload=13957]: Uploading segments... Aug 7 23:12:53.310488747 (13h33m27.825591336s) [upload=13957]: Got next segment. Inline: true Aug 7 23:12:53.310497967 (13h33m27.825600548s) [upload=13957]: Waiting for error group managing segments... Aug 7 23:12:53.310564975 (13h33m27.825667562s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:53.310572430 (13h33m27.825675017s) [upload=13957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:53.310576051 (13h33m27.825678628s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:23.280527765 (13h33m57.79563034s) [upload=13957]: Done uploading segments... Aug 7 23:13:23.280561185 (13h33m57.795663762s) [upload=13957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:50.810735096 (13h34m25.325837673s) [upload=13972]: Starting upload Aug 7 23:13:50.810789852 (13h34m25.325892437s) [upload=13972]: Done starting upload Aug 7 23:13:50.810850987 (13h34m25.325953556s) [upload=13972]: Uploading segments... Aug 7 23:13:50.810923216 (13h34m25.326025797s) [upload=13972]: Got next segment. Inline: true Aug 7 23:13:50.810928523 (13h34m25.326031096s) [upload=13972]: Waiting for error group managing segments... Aug 7 23:13:50.810970440 (13h34m25.326073016s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:50.810974710 (13h34m25.326077286s) [upload=13972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:50.810978274 (13h34m25.32608085s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:50.866577223 (13h34m25.3816798s) [upload=13972]: Done uploading segments... Aug 7 23:13:50.866594654 (13h34m25.38169723s) [upload=13972]: Upload finished. err: <nil> Aug 7 23:14:54.225692687 (13h35m28.740795254s) [upload=13986]: Starting upload Aug 7 23:14:54.225785134 (13h35m28.740887721s) [upload=13986]: Done starting upload Aug 7 23:14:54.225870963 (13h35m28.740973529s) [upload=13986]: Uploading segments... Aug 7 23:14:54.225923809 (13h35m28.741026386s) [upload=13986]: Got next segment. Inline: true Aug 7 23:14:54.225928631 (13h35m28.741031207s) [upload=13986]: Waiting for error group managing segments... Aug 7 23:14:54.225969273 (13h35m28.741071842s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:54.225973734 (13h35m28.7410763s) [upload=13986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:54.225976991 (13h35m28.741079566s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:24.191802298 (13h35m58.706904867s) [upload=13986]: Done uploading segments... Aug 7 23:15:24.191822370 (13h35m58.70692494s) [upload=13986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:24.446619359 (13h35m58.961721924s) [upload=13992]: Starting upload Aug 7 23:15:24.446725754 (13h35m58.961828338s) [upload=13992]: Done starting upload Aug 7 23:15:24.446812887 (13h35m58.961915463s) [upload=13992]: Uploading segments... Aug 7 23:15:24.446871932 (13h35m58.961974519s) [upload=13992]: Got next segment. Inline: true Aug 7 23:15:24.446878746 (13h35m58.961981315s) [upload=13992]: Waiting for error group managing segments... Aug 7 23:15:24.446939653 (13h35m58.962042222s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:24.446945288 (13h35m58.962047863s) [upload=13992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:24.446950467 (13h35m58.96205303s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:54.414392139 (13h36m28.929494705s) [upload=13992]: Done uploading segments... Aug 7 23:15:54.414412085 (13h36m28.929514651s) [upload=13992]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:32.150263694 (13h37m6.665366261s) [upload=14009]: Starting upload Aug 7 23:16:32.150345034 (13h37m6.665447612s) [upload=14009]: Done starting upload Aug 7 23:16:32.150402507 (13h37m6.665505071s) [upload=14009]: Uploading segments... Aug 7 23:16:32.150453597 (13h37m6.665556176s) [upload=14009]: Got next segment. Inline: true Aug 7 23:16:32.150458385 (13h37m6.665560958s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:16:32.150501065 (13h37m6.665603641s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:32.150505833 (13h37m6.66560841s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:32.150509597 (13h37m6.665612174s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:02.044102103 (13h37m36.55920467s) [upload=14009]: Done uploading segments... Aug 7 23:17:02.044123177 (13h37m36.559225743s) [upload=14009]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001002316s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 766.009337ms FAIL (2025-08-07 23:16:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 766.030819ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XX.XXX] duration: 1m0.001744903s FAIL (2025-08-07 23:16:56 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.016517629 (13h33m55.531620211s) [noise=true , address=104.36.82.43:4837 , piece_id=TCE3IJZU7ZA4Z5ONPBDQANINXZXA3BHKORRZGBB7AN6WZULACQ5A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.016800481 (13h33m55.531903059s) [noise=true , address=104.36.82.60:4365 , piece_id=F5LBKY4MEJI5AA26CHAHYT7D47B5SJU2SVBVHUMKRPKOH2YGLFRQ, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.017085412 (13h33m55.532187996s) [noise=true , address=104.36.82.9:4009 , piece_id=WSQW7NZSRPNYA4AM4T2A5T6KUWEHJJ47E6D66Y4S76RIX2KY7TMQ, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.017414361 (13h33m55.532516947s) [noise=true , address=104.36.82.47:4520 , piece_id=TS2GSEHDUCPNVAJJGGO4HBJQYAY4MUWN2LK3B7EFI5IJ7BHUP5HQ, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.017783045 (13h33m55.532885624s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.018198306 (13h33m55.533300869s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Uploading piece... Aug 7 23:13:21.099506775 (13h33m55.614609341s) [upload=13963]: Next returned nil segment Aug 7 23:13:21.099521270 (13h33m55.614623846s) [upload=13963]: Waiting for error group managing segments... Aug 7 23:13:21.104019028 (13h33m55.619121597s) [noise=true , address=45.91.116.5:4305 , piece_id=V2T7C5TPEPR4KDVZI67JO2DIIQGXNTCLOBEOOIZYGQYN4TB5UCKQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.109200706 (13h33m55.624303283s) [noise=true , address=104.36.82.17:4352 , piece_id=2FNMB25WSQMNVTAP7UCNXFJQJLF7VXVWSVOQLN2CTF4N6GQUQAGA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.110092715 (13h33m55.625195286s) [noise=true , address=104.36.82.43:4837 , piece_id=TCE3IJZU7ZA4Z5ONPBDQANINXZXA3BHKORRZGBB7AN6WZULACQ5A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.111990406 (13h33m55.627092973s) [noise=true , address=104.36.82.62:4435 , piece_id=3XPVD4VZ43PTPELTHNJHSMYCD4CIZGG3DBDBXDG247J22HYT6O2Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.112988613 (13h33m55.628091177s) [noise=true , address=104.36.82.53:4121 , piece_id=I3ZRDOT7YBFFZKRJHGQSJDMR3IEMZD754NFN6YWGVBO2ULYHOK4A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.113711920 (13h33m55.628814482s) [noise=true , address=104.36.82.2:4752 , piece_id=M5PDJWFGBNB4ELYBCGVZUXS5ELWJ46HGJ43IULWOUM2GEMJC6C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114233050 (13h33m55.629335628s) [noise=true , address=104.36.82.38:4712 , piece_id=6G4TGYY3JILWXGGVFMLX3BO77QRHM4OXVXJNL3YMAD53EU55MZXA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.114540807 (13h33m55.629643369s) [noise=true , address=45.91.116.8:4053 , piece_id=AB4MH3EBI2HTBZH6UZXJXHWPP7SK6TF6QAE6NHGVIJ363VOY3DVQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.122184649 (13h33m55.637287217s) [noise=true , address=104.36.82.44:4892 , piece_id=FXH2ZZHACDLSQCGJ4NBVVGLYBDDKA5J7KXT4RKUWJSBR2FOT7JUA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.123127372 (13h33m55.638229942s) [noise=true , address=104.36.82.47:4520 , piece_id=TS2GSEHDUCPNVAJJGGO4HBJQYAY4MUWN2LK3B7EFI5IJ7BHUP5HQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.125615705 (13h33m55.640718274s) [noise=true , address=104.36.82.60:4365 , piece_id=F5LBKY4MEJI5AA26CHAHYT7D47B5SJU2SVBVHUMKRPKOH2YGLFRQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.129141651 (13h33m55.644244214s) [noise=true , address=104.36.82.60:4357 , piece_id=UTRV3ECM4VKIMVI25QOD3GA7MLBO247ZDXYL7ZOPLOMM2RLSU6VQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.135781424 (13h33m55.650883986s) [noise=true , address=104.36.82.5:4844 , piece_id=REIROBOO56LH63HN6GTX237VQ4KYEACO2LXJGQ4KWPC2VQVEIPFA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145947229 (13h33m55.661049806s) [noise=true , address=104.36.82.12:4167 , piece_id=WJNPLTBMBLD7Z2ECP5SSJPDTHH7TX6U24T2OKTS7NUYIYN3C3C3A, seg_pos={0 0}, upload=13963]: Done uploading piece. err: <nil> Aug 7 23:13:21.145973097 (13h33m55.661075674s) [seg_pos={0 0}, upload=13963]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:21.146446342 (13h33m55.661548919s) [noise=true , address=104.36.82.39:4752 , piece_id=XXGMFHJX2IRCNBOYINHMKCJT5YS2Y72W6PL2DJQORRSHJ4OYFBCQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qoB7SaK4p6GwyHAKnheGGuMvQFzA3cerZL5nx7KnpazomyCTT): protocol: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32866->104.36.82.39:4752: use of closed network connection Aug 7 23:13:21.146527095 (13h33m55.661629665s) [noise=true , address=79.127.205.233:4629 , piece_id=JO4HOXAE4TUR44R55JVMESPMDHZVPRZUB2UI4RSPBXZWYPWUYO6Q, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fCtaroi1DDvaKo4kVVMLPpwDiLWwz377LEs7pyh3BHhvkiifX): protocol: read tcp 159.223.143.39:50930->79.127.205.233:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146720823 (13h33m55.661823393s) [noise=true , address=104.36.82.9:4009 , piece_id=WSQW7NZSRPNYA4AM4T2A5T6KUWEHJJ47E6D66Y4S76RIX2KY7TMQ, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb): protocol: read tcp 159.223.143.39:49842->104.36.82.9:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:21.146939426 (13h33m55.662041993s) [noise=true , address=64.49.13.154:4936 , piece_id=CP354MRPGNHBGBGMDDXCQWV4HMXLDE2BX5N3SDRI25757TH7LPJA, seg_pos={0 0}, upload=13963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J4nbkWW5oM17EbRwB53vgPzEiK8dEXM2tR2TWGhJFqEja9RHi): protocol: read tcp 159.223.143.39:39690->64.49.13.154:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:13:21.146954418 (13h33m55.662056994s) [seg_pos={0 0}, upload=13963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:21.147046940 (13h33m55.662149506s) [upload=13963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:21.147051626 (13h33m55.662154193s) [upload=13963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:21.197764982 (13h33m55.712867549s) [upload=13963]: Done uploading segments... Aug 7 23:13:21.197785171 (13h33m55.712887737s) [upload=13963]: Upload finished. err: <nil> Aug 7 23:13:52.912177255 (13h34m27.427279825s) [upload=13973]: Starting upload Aug 7 23:13:52.912245188 (13h34m27.427347764s) [upload=13973]: Done starting upload Aug 7 23:13:52.912322905 (13h34m27.427425471s) [upload=13973]: Uploading segments... Aug 7 23:13:52.912741963 (13h34m27.427844541s) [upload=13973]: Got next segment. Inline: false Aug 7 23:13:52.912787686 (13h34m27.427890271s) [upload=13973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:52.912794152 (13h34m27.427896731s) [upload=13973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:52.879876232 (13h35m27.394978808s) [upload=13973]: Next segment err: context deadline exceeded Aug 7 23:14:52.879935785 (13h35m27.395038353s) [upload=13973]: Done uploading segments... Aug 7 23:14:52.879952894 (13h35m27.395055462s) [upload=13973]: Upload finished. err: context deadline exceeded Aug 7 23:14:56.273083754 (13h35m30.78818632s) [upload=13987]: Starting upload Aug 7 23:14:56.273138206 (13h35m30.788240791s) [upload=13987]: Done starting upload Aug 7 23:14:56.273487156 (13h35m30.788589726s) [upload=13987]: Uploading segments... Aug 7 23:14:56.273528218 (13h35m30.788630804s) [upload=13987]: Got next segment. Inline: false Aug 7 23:14:56.273572431 (13h35m30.788675008s) [upload=13987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.273577073 (13h35m30.788679642s) [upload=13987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:56.244079248 (13h36m30.759181825s) [upload=13987]: Next segment err: context deadline exceeded Aug 7 23:15:56.244117923 (13h36m30.7592205s) [upload=13987]: Done uploading segments... Aug 7 23:15:56.244134836 (13h36m30.759237404s) [upload=13987]: Upload finished. err: context deadline exceeded Aug 7 23:15:56.354114004 (13h36m30.869216575s) [upload=13999]: Starting upload Aug 7 23:15:56.354173464 (13h36m30.869276042s) [upload=13999]: Done starting upload Aug 7 23:15:56.354235645 (13h36m30.869338234s) [upload=13999]: Uploading segments... Aug 7 23:15:56.354444202 (13h36m30.869546778s) [upload=13999]: Got next segment. Inline: false Aug 7 23:15:56.354481122 (13h36m30.869583698s) [upload=13999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:56.354484796 (13h36m30.869587372s) [upload=13999]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:56.323532882 (13h37m30.838635479s) [upload=13999]: Next segment err: context deadline exceeded Aug 7 23:16:56.323738594 (13h37m30.838841171s) [upload=13999]: Done uploading segments... Aug 7 23:16:56.323774957 (13h37m30.838877535s) [upload=13999]: Upload finished. err: context deadline exceeded Duration: 1m0.001762028s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................X..XXXX] duration: 30.000484798s FAIL (2025-08-07 23:16:25 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000507897s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX.X.] duration: 325.21077ms FAIL (2025-08-07 23:16:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 325.241641ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 1.755487299s FAIL (2025-08-07 23:16:11 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.755525679s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.002009628s FAIL (2025-08-07 23:16:03 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.002042139s core uplink us1

Orbiter Check History

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)