Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXX.X....X.X.................................XXXXXXXX] duration: 30.000390171s FAIL (2025-08-07 22:17:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00041377s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.X.................................XXXXXX] duration: 1m0.001459857s FAIL (2025-08-07 22:17:39 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=12895]: Starting upload Aug 7 22:08:26.591770003 (12h29m1.106872567s) [upload=12895]: Done starting upload Aug 7 22:08:26.591871625 (12h29m1.106974202s) [upload=12895]: Uploading segments... Aug 7 22:08:26.591931014 (12h29m1.107033599s) [upload=12895]: Got next segment. Inline: true Aug 7 22:08:26.591936202 (12h29m1.107038768s) [upload=12895]: Waiting for error group managing segments... Aug 7 22:08:26.591990877 (12h29m1.107093457s) [upload=12895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:26.591995259 (12h29m1.107097828s) [upload=12895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:26.591998403 (12h29m1.107100969s) [upload=12895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:26.672830843 (12h29m1.187933663s) [upload=12895]: Done uploading segments... Aug 7 22:08:26.672861348 (12h29m1.187963926s) [upload=12895]: Upload finished. err: <nil> Aug 7 22:08:55.672477854 (12h29m30.187580437s) [upload=12903]: Starting upload Aug 7 22:08:55.672581371 (12h29m30.187684016s) [upload=12903]: Done starting upload Aug 7 22:08:55.672796946 (12h29m30.187899524s) [upload=12903]: Uploading segments... Aug 7 22:08:55.672988832 (12h29m30.188091398s) [upload=12903]: Got next segment. Inline: true Aug 7 22:08:55.673020300 (12h29m30.188122877s) [upload=12903]: Waiting for error group managing segments... Aug 7 22:08:55.673111302 (12h29m30.188213889s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:55.673117396 (12h29m30.188219973s) [upload=12903]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:55.673121451 (12h29m30.188224017s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:55.748990886 (12h29m30.264093475s) [upload=12903]: Done uploading segments... Aug 7 22:08:55.749011264 (12h29m30.264113844s) [upload=12903]: Upload finished. err: <nil> Aug 7 22:09:33.459458247 (12h30m7.974560823s) [upload=12913]: Starting upload Aug 7 22:09:33.459560333 (12h30m7.974662911s) [upload=12913]: Done starting upload Aug 7 22:09:33.459666234 (12h30m7.974768811s) [upload=12913]: Uploading segments... Aug 7 22:09:33.459731513 (12h30m7.974834099s) [upload=12913]: Got next segment. Inline: true Aug 7 22:09:33.459737908 (12h30m7.974840491s) [upload=12913]: Waiting for error group managing segments... Aug 7 22:09:33.459810629 (12h30m7.974913195s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:33.459816521 (12h30m7.974919097s) [upload=12913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:33.459819752 (12h30m7.974922328s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:33.522052292 (12h30m8.037154862s) [upload=12913]: Done uploading segments... Aug 7 22:09:33.522072074 (12h30m8.037174651s) [upload=12913]: Upload finished. err: <nil> Aug 7 22:10:20.772089130 (12h30m55.287191696s) [upload=12928]: Starting upload Aug 7 22:10:20.772161634 (12h30m55.28726421s) [upload=12928]: Done starting upload Aug 7 22:10:20.772231905 (12h30m55.287334472s) [upload=12928]: Uploading segments... Aug 7 22:10:20.772284367 (12h30m55.287386947s) [upload=12928]: Got next segment. Inline: true Aug 7 22:10:20.772289092 (12h30m55.287391658s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:10:20.772344988 (12h30m55.287447554s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:20.772349640 (12h30m55.287452217s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:20.772352965 (12h30m55.287455541s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:20.857003207 (12h30m55.372105787s) [upload=12928]: Done uploading segments... Aug 7 22:10:20.857032212 (12h30m55.372134783s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:11:23.331604570 (12h31m57.846707137s) [upload=12944]: Starting upload Aug 7 22:11:23.331668435 (12h31m57.846771012s) [upload=12944]: Done starting upload Aug 7 22:11:23.331728315 (12h31m57.846830881s) [upload=12944]: Uploading segments... Aug 7 22:11:23.331766773 (12h31m57.846869355s) [upload=12944]: Got next segment. Inline: true Aug 7 22:11:23.331771532 (12h31m57.846874108s) [upload=12944]: Waiting for error group managing segments... Aug 7 22:11:23.331812523 (12h31m57.846915099s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:23.331817222 (12h31m57.846919788s) [upload=12944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:23.331820650 (12h31m57.846923226s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:23.301325187 (12h32m57.816427773s) [upload=12944]: Done uploading segments... Aug 7 22:12:23.301350364 (12h32m57.816452947s) [upload=12944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:33.831933438 (12h33m8.34703602s) [upload=12965]: Starting upload Aug 7 22:12:33.832018368 (12h33m8.34712093s) [upload=12965]: Done starting upload Aug 7 22:12:33.832086961 (12h33m8.347189539s) [upload=12965]: Uploading segments... Aug 7 22:12:33.832135510 (12h33m8.347238093s) [upload=12965]: Got next segment. Inline: true Aug 7 22:12:33.832141249 (12h33m8.347243826s) [upload=12965]: Waiting for error group managing segments... Aug 7 22:12:33.832194093 (12h33m8.347296674s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:33.832199847 (12h33m8.347302426s) [upload=12965]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:33.832205088 (12h33m8.347307666s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:33.802839016 (12h34m8.317941593s) [upload=12965]: Done uploading segments... Aug 7 22:13:33.802913313 (12h34m8.31801589s) [upload=12965]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.913275619 (12h34m8.428378249s) [upload=12977]: Starting upload Aug 7 22:13:33.913398357 (12h34m8.428500941s) [upload=12977]: Done starting upload Aug 7 22:13:33.913496779 (12h34m8.428599348s) [upload=12977]: Uploading segments... Aug 7 22:13:33.913568818 (12h34m8.428671424s) [upload=12977]: Got next segment. Inline: true Aug 7 22:13:33.913576485 (12h34m8.428679072s) [upload=12977]: Waiting for error group managing segments... Aug 7 22:13:33.913646668 (12h34m8.428749234s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.913652247 (12h34m8.428754834s) [upload=12977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.913655814 (12h34m8.428758404s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.874545239 (12h35m8.389647805s) [upload=12977]: Done uploading segments... Aug 7 22:14:33.874575517 (12h35m8.389678082s) [upload=12977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:38.961368058 (12h35m13.476470627s) [upload=12995]: Starting upload Aug 7 22:14:38.961455294 (12h35m13.476557856s) [upload=12995]: Done starting upload Aug 7 22:14:38.961527233 (12h35m13.476629807s) [upload=12995]: Uploading segments... Aug 7 22:14:38.961574411 (12h35m13.476676997s) [upload=12995]: Got next segment. Inline: true Aug 7 22:14:38.961580546 (12h35m13.47668312s) [upload=12995]: Waiting for error group managing segments... Aug 7 22:14:38.961644878 (12h35m13.476747457s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:38.961650553 (12h35m13.476753133s) [upload=12995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:38.961655154 (12h35m13.476757734s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:38.926876179 (12h36m13.441978762s) [upload=12995]: Done uploading segments... Aug 7 22:15:38.926898754 (12h36m13.442001334s) [upload=12995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:39.023006560 (12h36m13.538109126s) [upload=13009]: Starting upload Aug 7 22:15:39.023159911 (12h36m13.538262487s) [upload=13009]: Done starting upload Aug 7 22:15:39.023273936 (12h36m13.538376511s) [upload=13009]: Uploading segments... Aug 7 22:15:39.023358183 (12h36m13.538460767s) [upload=13009]: Got next segment. Inline: true Aug 7 22:15:39.023363552 (12h36m13.538466129s) [upload=13009]: Waiting for error group managing segments... Aug 7 22:15:39.023458469 (12h36m13.538561049s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:39.023464439 (12h36m13.538567019s) [upload=13009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:39.023491067 (12h36m13.53859365s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:38.991189889 (12h37m13.506292477s) [upload=13009]: Done uploading segments... Aug 7 22:16:38.991214272 (12h37m13.506316848s) [upload=13009]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:39.116496595 (12h37m13.631599182s) [upload=13020]: Starting upload Aug 7 22:16:39.116578790 (12h37m13.631681366s) [upload=13020]: Done starting upload Aug 7 22:16:39.116734512 (12h37m13.631837083s) [upload=13020]: Uploading segments... Aug 7 22:16:39.116797388 (12h37m13.631899986s) [upload=13020]: Got next segment. Inline: true Aug 7 22:16:39.116803839 (12h37m13.631906421s) [upload=13020]: Waiting for error group managing segments... Aug 7 22:16:39.116855539 (12h37m13.631958104s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:39.116860739 (12h37m13.631963318s) [upload=13020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:39.116864309 (12h37m13.631966875s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:39.087205419 (12h38m13.602307996s) [upload=13020]: Done uploading segments... Aug 7 22:17:39.087240451 (12h38m13.602343028s) [upload=13020]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001478387s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [...XX..XX................................................X..] duration: 55.866623ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:17:30 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 55.890627ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXX.....X..........................XXXX] duration: 1m0.000672225s FAIL (2025-08-07 22:17:26 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== .36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085855643 (12h31m52.600958229s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085941726 (12h31m52.601044291s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.086348257 (12h31m52.601450834s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.086383562 (12h31m52.601486144s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.087350241 (12h31m52.602452811s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.088850336 (12h31m52.603952921s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.178629350 (12h31m52.693731915s) [upload=12943]: Next returned nil segment Aug 7 22:11:18.178647680 (12h31m52.693750267s) [upload=12943]: Waiting for error group managing segments... Aug 7 22:11:18.197176897 (12h31m52.712279474s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.201535099 (12h31m52.716637672s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.212513462 (12h31m52.727616038s) [noise=true , address=104.36.82.47:4552 , piece_id=NJTH3OJMQQBV6PEADF4IFYZ3U7QZHJIU774CEPVY5BBBXI6V53QA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.213955699 (12h31m52.729058282s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.214449276 (12h31m52.729551846s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.217895630 (12h31m52.732998217s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.218622391 (12h31m52.733724974s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.219619423 (12h31m52.734722004s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.227715032 (12h31m52.742817609s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.228516300 (12h31m52.743618876s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.241530379 (12h31m52.756632941s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.251935797 (12h31m52.767038375s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.277091280 (12h31m52.792193849s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291897936 (12h31m52.807000513s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291922152 (12h31m52.80702473s) [seg_pos={0 0}, upload=12943]: Segment reached optimal threshold of 14 pieces. Aug 7 22:11:18.292352474 (12h31m52.807455054s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:18.292749416 (12h31m52.807851996s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection Aug 7 22:11:18.293007428 (12h31m52.808110005s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: read tcp 159.223.143.39:34280->64.49.13.26:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293070269 (12h31m52.808172846s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: read tcp 159.223.143.39:46438->104.36.82.45:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293086568 (12h31m52.808189151s) [seg_pos={0 0}, upload=12943]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:11:18.293166574 (12h31m52.808269155s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:18.293172005 (12h31m52.808274588s) [upload=12943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:08.727892812 (12h34m43.242995378s) [upload=12943]: Done uploading segments... Aug 7 22:14:08.727919062 (12h34m43.243021627s) [upload=12943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:08.820786996 (12h34m43.335889564s) [upload=12987]: Starting upload Aug 7 22:14:08.820869466 (12h34m43.335972332s) [upload=12987]: Done starting upload Aug 7 22:14:08.820990812 (12h34m43.336093389s) [upload=12987]: Uploading segments... Aug 7 22:14:08.821180945 (12h34m43.336283523s) [upload=12987]: Got next segment. Inline: false Aug 7 22:14:08.821227735 (12h34m43.336330315s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:08.821233790 (12h34m43.336336478s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:08.788378642 (12h35m43.303481218s) [upload=12987]: Next segment err: context deadline exceeded Aug 7 22:15:08.788567212 (12h35m43.303669789s) [upload=12987]: Done uploading segments... Aug 7 22:15:08.788593806 (12h35m43.303696388s) [upload=12987]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.518494042 (12h35m57.033596616s) [upload=13004]: Starting upload Aug 7 22:15:22.518605977 (12h35m57.033708558s) [upload=13004]: Done starting upload Aug 7 22:15:22.518927572 (12h35m57.034030143s) [upload=13004]: Uploading segments... Aug 7 22:15:22.519083330 (12h35m57.034185915s) [upload=13004]: Got next segment. Inline: false Aug 7 22:15:22.519139330 (12h35m57.034241894s) [upload=13004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.519145674 (12h35m57.034248259s) [upload=13004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.486086322 (12h36m57.001188918s) [upload=13004]: Next segment err: context deadline exceeded Aug 7 22:16:22.486315849 (12h36m57.00141843s) [upload=13004]: Done uploading segments... Aug 7 22:16:22.486353471 (12h36m57.001456038s) [upload=13004]: Upload finished. err: context deadline exceeded Aug 7 22:16:26.977501497 (12h37m1.492604079s) [upload=13017]: Starting upload Aug 7 22:16:26.977634110 (12h37m1.492736693s) [upload=13017]: Done starting upload Aug 7 22:16:26.977724277 (12h37m1.492826848s) [upload=13017]: Uploading segments... Aug 7 22:16:26.978074717 (12h37m1.493177287s) [upload=13017]: Got next segment. Inline: false Aug 7 22:16:26.978159060 (12h37m1.493261638s) [upload=13017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:26.978164291 (12h37m1.493266867s) [upload=13017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:26.948529899 (12h38m1.463632483s) [upload=13017]: Next segment err: context deadline exceeded Aug 7 22:17:26.948553603 (12h38m1.463656182s) [upload=13017]: Done uploading segments... Aug 7 22:17:26.948580927 (12h38m1.463683497s) [upload=13017]: Upload finished. err: context deadline exceeded Duration: 1m0.000695019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX....XX................................XXXXXXXXX] duration: 30.000903313s FAIL (2025-08-07 22:17:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000915742s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XX.X..X..X..........................XXXXXXXX] duration: 30.001104272s FAIL (2025-08-07 22:17:15 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00112804s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........................................X......] duration: 1.485612295s FAIL (2025-08-07 22:17:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 1.485642599s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 273.887577ms FAIL (2025-08-07 22:17:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 273.910862ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX....XX................................XXXXXXXX] duration: 30.001436169s FAIL (2025-08-07 22:16:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001457893s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XX...X.............................XXXXXX] duration: 1m0.001494691s FAIL (2025-08-07 22:16:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001510109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XX...XXX...............................XXXXXXX] duration: 30.001492821s FAIL (2025-08-07 22:16:49 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9.304915875 (12h29m43.820018452s) [upload=12910]: Done starting upload Aug 7 22:09:09.305006211 (12h29m43.820108779s) [upload=12910]: Uploading segments... Aug 7 22:09:09.305087545 (12h29m43.820190134s) [upload=12910]: Got next segment. Inline: true Aug 7 22:09:09.305095349 (12h29m43.820197936s) [upload=12910]: Waiting for error group managing segments... Aug 7 22:09:09.305151615 (12h29m43.820254177s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:09.305157403 (12h29m43.820259965s) [upload=12910]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:09.305160723 (12h29m43.820263285s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:09.641838505 (12h29m44.156941082s) [upload=12910]: Done uploading segments... Aug 7 22:09:09.641856719 (12h29m44.156959297s) [upload=12910]: Upload finished. err: <nil> Aug 7 22:09:35.598760483 (12h30m10.113863059s) [upload=12914]: Starting upload Aug 7 22:09:35.598830860 (12h30m10.113933439s) [upload=12914]: Done starting upload Aug 7 22:09:35.598909885 (12h30m10.11401245s) [upload=12914]: Uploading segments... Aug 7 22:09:35.598948468 (12h30m10.114051051s) [upload=12914]: Got next segment. Inline: true Aug 7 22:09:35.598952824 (12h30m10.114055403s) [upload=12914]: Waiting for error group managing segments... Aug 7 22:09:35.599006219 (12h30m10.114108795s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:35.599011436 (12h30m10.114114015s) [upload=12914]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:35.599014692 (12h30m10.114117272s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:35.668939111 (12h30m10.184041679s) [upload=12914]: Done uploading segments... Aug 7 22:09:35.668959039 (12h30m10.184061615s) [upload=12914]: Upload finished. err: <nil> Aug 7 22:10:18.242162305 (12h30m52.757264891s) [upload=12926]: Starting upload Aug 7 22:10:18.242275276 (12h30m52.75737786s) [upload=12926]: Done starting upload Aug 7 22:10:18.242410011 (12h30m52.757512577s) [upload=12926]: Uploading segments... Aug 7 22:10:18.242471376 (12h30m52.757573961s) [upload=12926]: Got next segment. Inline: true Aug 7 22:10:18.242478056 (12h30m52.757580635s) [upload=12926]: Waiting for error group managing segments... Aug 7 22:10:18.242535768 (12h30m52.757638347s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:18.242540806 (12h30m52.757643383s) [upload=12926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:18.242544109 (12h30m52.757646685s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:18.309606842 (12h30m52.824709422s) [upload=12926]: Done uploading segments... Aug 7 22:10:18.309625536 (12h30m52.824728119s) [upload=12926]: Upload finished. err: <nil> Aug 7 22:11:28.337393719 (12h32m2.852496289s) [upload=12946]: Starting upload Aug 7 22:11:28.337459886 (12h32m2.852562466s) [upload=12946]: Done starting upload Aug 7 22:11:28.337521996 (12h32m2.852624566s) [upload=12946]: Uploading segments... Aug 7 22:11:28.337560362 (12h32m2.852662948s) [upload=12946]: Got next segment. Inline: true Aug 7 22:11:28.337565059 (12h32m2.852667634s) [upload=12946]: Waiting for error group managing segments... Aug 7 22:11:28.337604389 (12h32m2.852706965s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:28.337608838 (12h32m2.852711414s) [upload=12946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:28.337612338 (12h32m2.852714915s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:58.307179904 (12h32m32.82228248s) [upload=12946]: Done uploading segments... Aug 7 22:11:58.307198958 (12h32m32.822301534s) [upload=12946]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:21.223007637 (12h32m55.738110218s) [upload=12959]: Starting upload Aug 7 22:12:21.223121202 (12h32m55.738223785s) [upload=12959]: Done starting upload Aug 7 22:12:21.223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:19.727995115 (12h36m54.243097692s) [upload=13016]: Starting upload Aug 7 22:16:19.728064428 (12h36m54.243167004s) [upload=13016]: Done starting upload Aug 7 22:16:19.728134488 (12h36m54.243237051s) [upload=13016]: Uploading segments... Aug 7 22:16:19.728182154 (12h36m54.243284736s) [upload=13016]: Got next segment. Inline: true Aug 7 22:16:19.728186380 (12h36m54.243288955s) [upload=13016]: Waiting for error group managing segments... Aug 7 22:16:19.728223320 (12h36m54.243325899s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:19.728227150 (12h36m54.243329727s) [upload=13016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:19.728229527 (12h36m54.243332103s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:49.691229995 (12h37m24.206332575s) [upload=13016]: Done uploading segments... Aug 7 22:16:49.691252411 (12h37m24.206354988s) [upload=13016]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001517839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.........................................X.....] duration: 1.334053642s FAIL (2025-08-07 22:16:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 1.334081674s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 307.831654ms FAIL (2025-08-07 22:16:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 307.855744ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.X.................................XXXXX] duration: 1m0.001334958s FAIL (2025-08-07 22:16:38 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8:03.216043710 (12h28m37.731146297s) [upload=12889]: Starting upload Aug 7 22:08:03.216136680 (12h28m37.731239259s) [upload=12889]: Done starting upload Aug 7 22:08:03.216283833 (12h28m37.731386404s) [upload=12889]: Uploading segments... Aug 7 22:08:03.216375553 (12h28m37.73147814s) [upload=12889]: Got next segment. Inline: true Aug 7 22:08:03.216382176 (12h28m37.731484755s) [upload=12889]: Waiting for error group managing segments... Aug 7 22:08:03.216448592 (12h28m37.73155116s) [upload=12889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:03.216454041 (12h28m37.731556619s) [upload=12889]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:03.216457480 (12h28m37.731560058s) [upload=12889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:03.293854136 (12h28m37.808956726s) [upload=12889]: Done uploading segments... Aug 7 22:08:03.293877099 (12h28m37.80897968s) [upload=12889]: Upload finished. err: <nil> Aug 7 22:08:26.591681271 (12h29m1.106783851s) [upload=12895]: Starting upload Aug 7 22:08:26.591770003 (12h29m1.106872567s) [upload=12895]: Done starting upload Aug 7 22:08:26.591871625 (12h29m1.106974202s) [upload=12895]: Uploading segments... Aug 7 22:08:26.591931014 (12h29m1.107033599s) [upload=12895]: Got next segment. Inline: true Aug 7 22:08:26.591936202 (12h29m1.107038768s) [upload=12895]: Waiting for error group managing segments... Aug 7 22:08:26.591990877 (12h29m1.107093457s) [upload=12895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:26.591995259 (12h29m1.107097828s) [upload=12895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:26.591998403 (12h29m1.107100969s) [upload=12895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:26.672830843 (12h29m1.187933663s) [upload=12895]: Done uploading segments... Aug 7 22:08:26.672861348 (12h29m1.187963926s) [upload=12895]: Upload finished. err: <nil> Aug 7 22:08:55.672477854 (12h29m30.187580437s) [upload=12903]: Starting upload Aug 7 22:08:55.672581371 (12h29m30.187684016s) [upload=12903]: Done starting upload Aug 7 22:08:55.672796946 (12h29m30.187899524s) [upload=12903]: Uploading segments... Aug 7 22:08:55.672988832 (12h29m30.188091398s) [upload=12903]: Got next segment. Inline: true Aug 7 22:08:55.673020300 (12h29m30.188122877s) [upload=12903]: Waiting for error group managing segments... Aug 7 22:08:55.673111302 (12h29m30.188213889s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:55.673117396 (12h29m30.188219973s) [upload=12903]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:55.673121451 (12h29m30.188224017s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:55.748990886 (12h29m30.264093475s) [upload=12903]: Done uploading segments... Aug 7 22:08:55.749011264 (12h29m30.264113844s) [upload=12903]: Upload finished. err: <nil> Aug 7 22:09:33.459458247 (12h30m7.974560823s) [upload=12913]: Starting upload Aug 7 22:09:33.459560333 (12h30m7.974662911s) [upload=12913]: Done starting upload Aug 7 22:09:33.459666234 (12h30m7.974768811s) [upload=12913]: Uploading segments... Aug 7 22:09:33.459731513 (12h30m7.974834099s) [upload=12913]: Got next segment. Inline: true Aug 7 22:09:33.459737908 (12h30m7.974840491s) [upload=12913]: Waiting for error group managing segments... Aug 7 22:09:33.459810629 (12h30m7.974913195s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:33.459816521 (12h30m7.974919097s) [upload=12913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:33.459819752 (12h30m7.974922328s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:33.522052292 (12h30m8.037154862s) [upload=12913]: Done uploading segments... Aug 7 22:09:33.522072074 (12h30m8.037174651s) [upload=12913]: Upload finished. err: <nil> Aug 7 22:10:20.772089130 (12h30m55.287191696s) [upload=12928]: Starting upload Aug 7 22:10:20.772161634 (12h30m55.28726421s) [upload=12928]: Done starting upload Aug 7 22:10:20.772231905 (12h30m55.287334472s) [upload=12928]: Uploading segments... Aug 7 22:10:20.772284367 (12h30m55.287386947s) [upload=12928]: Got next segment. Inline: true Aug 7 22:10:20.772289092 (12h30m55.287391658s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:10:20.772344988 (12h30m55.287447554s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:20.772349640 (12h30m55.287452217s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:20.772352965 (12h30m55.287455541s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:20.857003207 (12h30m55.372105787s) [upload=12928]: Done uploading segments... Aug 7 22:10:20.857032212 (12h30m55.372134783s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:11:23.331604570 (12h31m57.846707137s) [upload=12944]: Starting upload Aug 7 22:11:23.331668435 (12h31m57.846771012s) [upload=12944]: Done starting upload Aug 7 22:11:23.331728315 (12h31m57.846830881s) [upload=12944]: Uploading segments... Aug 7 22:11:23.331766773 (12h31m57.846869355s) [upload=12944]: Got next segment. Inline: true Aug 7 22:11:23.331771532 (12h31m57.846874108s) [upload=12944]: Waiting for error group managing segments... Aug 7 22:11:23.331812523 (12h31m57.846915099s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:23.331817222 (12h31m57.846919788s) [upload=12944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:23.331820650 (12h31m57.846923226s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:23.301325187 (12h32m57.816427773s) [upload=12944]: Done uploading segments... Aug 7 22:12:23.301350364 (12h32m57.816452947s) [upload=12944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:33.831933438 (12h33m8.34703602s) [upload=12965]: Starting upload Aug 7 22:12:33.832018368 (12h33m8.34712093s) [upload=12965]: Done starting upload Aug 7 22:12:33.832086961 (12h33m8.347189539s) [upload=12965]: Uploading segments... Aug 7 22:12:33.832135510 (12h33m8.347238093s) [upload=12965]: Got next segment. Inline: true Aug 7 22:12:33.832141249 (12h33m8.347243826s) [upload=12965]: Waiting for error group managing segments... Aug 7 22:12:33.832194093 (12h33m8.347296674s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:33.832199847 (12h33m8.347302426s) [upload=12965]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:33.832205088 (12h33m8.347307666s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:33.802839016 (12h34m8.317941593s) [upload=12965]: Done uploading segments... Aug 7 22:13:33.802913313 (12h34m8.31801589s) [upload=12965]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.913275619 (12h34m8.428378249s) [upload=12977]: Starting upload Aug 7 22:13:33.913398357 (12h34m8.428500941s) [upload=12977]: Done starting upload Aug 7 22:13:33.913496779 (12h34m8.428599348s) [upload=12977]: Uploading segments... Aug 7 22:13:33.913568818 (12h34m8.428671424s) [upload=12977]: Got next segment. Inline: true Aug 7 22:13:33.913576485 (12h34m8.428679072s) [upload=12977]: Waiting for error group managing segments... Aug 7 22:13:33.913646668 (12h34m8.428749234s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.913652247 (12h34m8.428754834s) [upload=12977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.913655814 (12h34m8.428758404s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.874545239 (12h35m8.389647805s) [upload=12977]: Done uploading segments... Aug 7 22:14:33.874575517 (12h35m8.389678082s) [upload=12977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:38.961368058 (12h35m13.476470627s) [upload=12995]: Starting upload Aug 7 22:14:38.961455294 (12h35m13.476557856s) [upload=12995]: Done starting upload Aug 7 22:14:38.961527233 (12h35m13.476629807s) [upload=12995]: Uploading segments... Aug 7 22:14:38.961574411 (12h35m13.476676997s) [upload=12995]: Got next segment. Inline: true Aug 7 22:14:38.961580546 (12h35m13.47668312s) [upload=12995]: Waiting for error group managing segments... Aug 7 22:14:38.961644878 (12h35m13.476747457s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:38.961650553 (12h35m13.476753133s) [upload=12995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:38.961655154 (12h35m13.476757734s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:38.926876179 (12h36m13.441978762s) [upload=12995]: Done uploading segments... Aug 7 22:15:38.926898754 (12h36m13.442001334s) [upload=12995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:39.023006560 (12h36m13.538109126s) [upload=13009]: Starting upload Aug 7 22:15:39.023159911 (12h36m13.538262487s) [upload=13009]: Done starting upload Aug 7 22:15:39.023273936 (12h36m13.538376511s) [upload=13009]: Uploading segments... Aug 7 22:15:39.023358183 (12h36m13.538460767s) [upload=13009]: Got next segment. Inline: true Aug 7 22:15:39.023363552 (12h36m13.538466129s) [upload=13009]: Waiting for error group managing segments... Aug 7 22:15:39.023458469 (12h36m13.538561049s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:39.023464439 (12h36m13.538567019s) [upload=13009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:39.023491067 (12h36m13.53859365s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:38.991189889 (12h37m13.506292477s) [upload=13009]: Done uploading segments... Aug 7 22:16:38.991214272 (12h37m13.506316848s) [upload=13009]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001352085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXX.X....X.X.................................XXXXXXX] duration: 30.000424182s FAIL (2025-08-07 22:16:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000435497s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 29.032838ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:16:30 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 29.049598ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XX.X..X..X..........................XXXXXXX] duration: 30.00107959s FAIL (2025-08-07 22:16:25 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001105661s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXX.....X..........................XXX] duration: 1m0.001109947s FAIL (2025-08-07 22:16:22 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 11:18.084407796 (12h31m52.59951038s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084922562 (12h31m52.600025148s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084938723 (12h31m52.6000413s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085354116 (12h31m52.600456699s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085438941 (12h31m52.600541507s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085855643 (12h31m52.600958229s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085941726 (12h31m52.601044291s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.086348257 (12h31m52.601450834s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.086383562 (12h31m52.601486144s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.087350241 (12h31m52.602452811s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.088850336 (12h31m52.603952921s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.178629350 (12h31m52.693731915s) [upload=12943]: Next returned nil segment Aug 7 22:11:18.178647680 (12h31m52.693750267s) [upload=12943]: Waiting for error group managing segments... Aug 7 22:11:18.197176897 (12h31m52.712279474s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.201535099 (12h31m52.716637672s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.212513462 (12h31m52.727616038s) [noise=true , address=104.36.82.47:4552 , piece_id=NJTH3OJMQQBV6PEADF4IFYZ3U7QZHJIU774CEPVY5BBBXI6V53QA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.213955699 (12h31m52.729058282s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.214449276 (12h31m52.729551846s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.217895630 (12h31m52.732998217s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.218622391 (12h31m52.733724974s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.219619423 (12h31m52.734722004s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.227715032 (12h31m52.742817609s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.228516300 (12h31m52.743618876s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.241530379 (12h31m52.756632941s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.251935797 (12h31m52.767038375s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.277091280 (12h31m52.792193849s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291897936 (12h31m52.807000513s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291922152 (12h31m52.80702473s) [seg_pos={0 0}, upload=12943]: Segment reached optimal threshold of 14 pieces. Aug 7 22:11:18.292352474 (12h31m52.807455054s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:18.292749416 (12h31m52.807851996s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection Aug 7 22:11:18.293007428 (12h31m52.808110005s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: read tcp 159.223.143.39:34280->64.49.13.26:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293070269 (12h31m52.808172846s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: read tcp 159.223.143.39:46438->104.36.82.45:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293086568 (12h31m52.808189151s) [seg_pos={0 0}, upload=12943]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:11:18.293166574 (12h31m52.808269155s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:18.293172005 (12h31m52.808274588s) [upload=12943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:08.727892812 (12h34m43.242995378s) [upload=12943]: Done uploading segments... Aug 7 22:14:08.727919062 (12h34m43.243021627s) [upload=12943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:08.820786996 (12h34m43.335889564s) [upload=12987]: Starting upload Aug 7 22:14:08.820869466 (12h34m43.335972332s) [upload=12987]: Done starting upload Aug 7 22:14:08.820990812 (12h34m43.336093389s) [upload=12987]: Uploading segments... Aug 7 22:14:08.821180945 (12h34m43.336283523s) [upload=12987]: Got next segment. Inline: false Aug 7 22:14:08.821227735 (12h34m43.336330315s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:08.821233790 (12h34m43.336336478s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:08.788378642 (12h35m43.303481218s) [upload=12987]: Next segment err: context deadline exceeded Aug 7 22:15:08.788567212 (12h35m43.303669789s) [upload=12987]: Done uploading segments... Aug 7 22:15:08.788593806 (12h35m43.303696388s) [upload=12987]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.518494042 (12h35m57.033596616s) [upload=13004]: Starting upload Aug 7 22:15:22.518605977 (12h35m57.033708558s) [upload=13004]: Done starting upload Aug 7 22:15:22.518927572 (12h35m57.034030143s) [upload=13004]: Uploading segments... Aug 7 22:15:22.519083330 (12h35m57.034185915s) [upload=13004]: Got next segment. Inline: false Aug 7 22:15:22.519139330 (12h35m57.034241894s) [upload=13004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.519145674 (12h35m57.034248259s) [upload=13004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.486086322 (12h36m57.001188918s) [upload=13004]: Next segment err: context deadline exceeded Aug 7 22:16:22.486315849 (12h36m57.00141843s) [upload=13004]: Done uploading segments... Aug 7 22:16:22.486353471 (12h36m57.001456038s) [upload=13004]: Upload finished. err: context deadline exceeded Duration: 1m0.001133688s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [....XX..XX................................................X.] duration: 45.002729ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:16:19 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 45.019696ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XX...XXX...............................XXXXXX] duration: 30.000632293s FAIL (2025-08-07 22:16:10 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=12896]: Starting upload Aug 7 22:08:28.430857402 (12h29m2.94595997s) [upload=12896]: Done starting upload Aug 7 22:08:28.430919754 (12h29m2.946022323s) [upload=12896]: Uploading segments... Aug 7 22:08:28.430957561 (12h29m2.946060143s) [upload=12896]: Got next segment. Inline: true Aug 7 22:08:28.430962783 (12h29m2.946065362s) [upload=12896]: Waiting for error group managing segments... Aug 7 22:08:28.431004484 (12h29m2.946107061s) [upload=12896]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:28.431008904 (12h29m2.946111481s) [upload=12896]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:28.431012134 (12h29m2.946114712s) [upload=12896]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:28.505132712 (12h29m3.020235288s) [upload=12896]: Done uploading segments... Aug 7 22:08:28.505162495 (12h29m3.020265065s) [upload=12896]: Upload finished. err: <nil> Aug 7 22:09:09.304833839 (12h29m43.819936409s) [upload=12910]: Starting upload Aug 7 22:09:09.304915875 (12h29m43.820018452s) [upload=12910]: Done starting upload Aug 7 22:09:09.305006211 (12h29m43.820108779s) [upload=12910]: Uploading segments... Aug 7 22:09:09.305087545 (12h29m43.820190134s) [upload=12910]: Got next segment. Inline: true Aug 7 22:09:09.305095349 (12h29m43.820197936s) [upload=12910]: Waiting for error group managing segments... Aug 7 22:09:09.305151615 (12h29m43.820254177s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:09.305157403 (12h29m43.820259965s) [upload=12910]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:09.305160723 (12h29m43.820263285s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:09.641838505 (12h29m44.156941082s) [upload=12910]: Done uploading segments... Aug 7 22:09:09.641856719 (12h29m44.156959297s) [upload=12910]: Upload finished. err: <nil> Aug 7 22:09:35.598760483 (12h30m10.113863059s) [upload=12914]: Starting upload Aug 7 22:09:35.598830860 (12h30m10.113933439s) [upload=12914]: Done starting upload Aug 7 22:09:35.598909885 (12h30m10.11401245s) [upload=12914]: Uploading segments... Aug 7 22:09:35.598948468 (12h30m10.114051051s) [upload=12914]: Got next segment. Inline: true Aug 7 22:09:35.598952824 (12h30m10.114055403s) [upload=12914]: Waiting for error group managing segments... Aug 7 22:09:35.599006219 (12h30m10.114108795s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:35.599011436 (12h30m10.114114015s) [upload=12914]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:35.599014692 (12h30m10.114117272s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:35.668939111 (12h30m10.184041679s) [upload=12914]: Done uploading segments... Aug 7 22:09:35.668959039 (12h30m10.184061615s) [upload=12914]: Upload finished. err: <nil> Aug 7 22:10:18.242162305 (12h30m52.757264891s) [upload=12926]: Starting upload Aug 7 22:10:18.242275276 (12h30m52.75737786s) [upload=12926]: Done starting upload Aug 7 22:10:18.242410011 (12h30m52.757512577s) [upload=12926]: Uploading segments... Aug 7 22:10:18.242471376 (12h30m52.757573961s) [upload=12926]: Got next segment. Inline: true Aug 7 22:10:18.242478056 (12h30m52.757580635s) [upload=12926]: Waiting for error group managing segments... Aug 7 22:10:18.242535768 (12h30m52.757638347s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:18.242540806 (12h30m52.757643383s) [upload=12926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:18.242544109 (12h30m52.757646685s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:18.309606842 (12h30m52.824709422s) [upload=12926]: Done uploading segments... Aug 7 22:10:18.309625536 (12h30m52.824728119s) [upload=12926]: Upload finished. err: <nil> Aug 7 22:11:28.337393719 (12h32m2.852496289s) [upload=12946]: Starting upload Aug 7 22:11:28.337459886 (12h32m2.852562466s) [upload=12946]: Done starting upload Aug 7 22:11:28.337521996 (12h32m2.852624566s) [upload=12946]: Uploading segments... Aug 7 22:11:28.337560362 (12h32m2.852662948s) [upload=12946]: Got next segment. Inline: true Aug 7 22:11:28.337565059 (12h32m2.852667634s) [upload=12946]: Waiting for error group managing segments... Aug 7 22:11:28.337604389 (12h32m2.852706965s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:28.337608838 (12h32m2.852711414s) [upload=12946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:28.337612338 (12h32m2.852714915s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:58.307179904 (12h32m32.82228248s) [upload=12946]: Done uploading segments... Aug 7 22:11:58.307198958 (12h32m32.822301534s) [upload=12946]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:21.223007637 (12h32m55.738110218s) [upload=12959]: Starting upload Aug 7 22:12:21.223121202 (12h32m55.738223785s) [upload=12959]: Done starting upload Aug 7 22:12:21.223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000655502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 464.584755ms FAIL (2025-08-07 22:16:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 464.61289ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.X....X.X.................................XXXXXX] duration: 30.001759021s FAIL (2025-08-07 22:16:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00178242s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XX...X.............................XXXXX] duration: 1m0.001582834s FAIL (2025-08-07 22:15:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001597032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX....XX................................XXXXXXX] duration: 30.000554918s FAIL (2025-08-07 22:15:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000570026s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........................................X....] duration: 1.152362904s FAIL (2025-08-07 22:15:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 1.15239566s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.X.................................XXXX] duration: 1m0.000517935s FAIL (2025-08-07 22:15:38 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== finished. err: <nil> Aug 7 22:06:54.132475466 (12h27m28.647578034s) [upload=12873]: Starting upload Aug 7 22:06:54.132547886 (12h27m28.647650462s) [upload=12873]: Done starting upload Aug 7 22:06:54.132786697 (12h27m28.647889262s) [upload=12873]: Uploading segments... Aug 7 22:06:54.132870099 (12h27m28.647972686s) [upload=12873]: Got next segment. Inline: true Aug 7 22:06:54.132877110 (12h27m28.647979693s) [upload=12873]: Waiting for error group managing segments... Aug 7 22:06:54.132948498 (12h27m28.648051076s) [upload=12873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:54.132953672 (12h27m28.648056256s) [upload=12873]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:54.132956651 (12h27m28.648059229s) [upload=12873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:54.200614549 (12h27m28.71571713s) [upload=12873]: Done uploading segments... Aug 7 22:06:54.200630357 (12h27m28.715732933s) [upload=12873]: Upload finished. err: <nil> Aug 7 22:08:03.216043710 (12h28m37.731146297s) [upload=12889]: Starting upload Aug 7 22:08:03.216136680 (12h28m37.731239259s) [upload=12889]: Done starting upload Aug 7 22:08:03.216283833 (12h28m37.731386404s) [upload=12889]: Uploading segments... Aug 7 22:08:03.216375553 (12h28m37.73147814s) [upload=12889]: Got next segment. Inline: true Aug 7 22:08:03.216382176 (12h28m37.731484755s) [upload=12889]: Waiting for error group managing segments... Aug 7 22:08:03.216448592 (12h28m37.73155116s) [upload=12889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:03.216454041 (12h28m37.731556619s) [upload=12889]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:03.216457480 (12h28m37.731560058s) [upload=12889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:03.293854136 (12h28m37.808956726s) [upload=12889]: Done uploading segments... Aug 7 22:08:03.293877099 (12h28m37.80897968s) [upload=12889]: Upload finished. err: <nil> Aug 7 22:08:26.591681271 (12h29m1.106783851s) [upload=12895]: Starting upload Aug 7 22:08:26.591770003 (12h29m1.106872567s) [upload=12895]: Done starting upload Aug 7 22:08:26.591871625 (12h29m1.106974202s) [upload=12895]: Uploading segments... Aug 7 22:08:26.591931014 (12h29m1.107033599s) [upload=12895]: Got next segment. Inline: true Aug 7 22:08:26.591936202 (12h29m1.107038768s) [upload=12895]: Waiting for error group managing segments... Aug 7 22:08:26.591990877 (12h29m1.107093457s) [upload=12895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:26.591995259 (12h29m1.107097828s) [upload=12895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:26.591998403 (12h29m1.107100969s) [upload=12895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:26.672830843 (12h29m1.187933663s) [upload=12895]: Done uploading segments... Aug 7 22:08:26.672861348 (12h29m1.187963926s) [upload=12895]: Upload finished. err: <nil> Aug 7 22:08:55.672477854 (12h29m30.187580437s) [upload=12903]: Starting upload Aug 7 22:08:55.672581371 (12h29m30.187684016s) [upload=12903]: Done starting upload Aug 7 22:08:55.672796946 (12h29m30.187899524s) [upload=12903]: Uploading segments... Aug 7 22:08:55.672988832 (12h29m30.188091398s) [upload=12903]: Got next segment. Inline: true Aug 7 22:08:55.673020300 (12h29m30.188122877s) [upload=12903]: Waiting for error group managing segments... Aug 7 22:08:55.673111302 (12h29m30.188213889s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:55.673117396 (12h29m30.188219973s) [upload=12903]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:55.673121451 (12h29m30.188224017s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:55.748990886 (12h29m30.264093475s) [upload=12903]: Done uploading segments... Aug 7 22:08:55.749011264 (12h29m30.264113844s) [upload=12903]: Upload finished. err: <nil> Aug 7 22:09:33.459458247 (12h30m7.974560823s) [upload=12913]: Starting upload Aug 7 22:09:33.459560333 (12h30m7.974662911s) [upload=12913]: Done starting upload Aug 7 22:09:33.459666234 (12h30m7.974768811s) [upload=12913]: Uploading segments... Aug 7 22:09:33.459731513 (12h30m7.974834099s) [upload=12913]: Got next segment. Inline: true Aug 7 22:09:33.459737908 (12h30m7.974840491s) [upload=12913]: Waiting for error group managing segments... Aug 7 22:09:33.459810629 (12h30m7.974913195s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:33.459816521 (12h30m7.974919097s) [upload=12913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:33.459819752 (12h30m7.974922328s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:33.522052292 (12h30m8.037154862s) [upload=12913]: Done uploading segments... Aug 7 22:09:33.522072074 (12h30m8.037174651s) [upload=12913]: Upload finished. err: <nil> Aug 7 22:10:20.772089130 (12h30m55.287191696s) [upload=12928]: Starting upload Aug 7 22:10:20.772161634 (12h30m55.28726421s) [upload=12928]: Done starting upload Aug 7 22:10:20.772231905 (12h30m55.287334472s) [upload=12928]: Uploading segments... Aug 7 22:10:20.772284367 (12h30m55.287386947s) [upload=12928]: Got next segment. Inline: true Aug 7 22:10:20.772289092 (12h30m55.287391658s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:10:20.772344988 (12h30m55.287447554s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:20.772349640 (12h30m55.287452217s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:20.772352965 (12h30m55.287455541s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:20.857003207 (12h30m55.372105787s) [upload=12928]: Done uploading segments... Aug 7 22:10:20.857032212 (12h30m55.372134783s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:11:23.331604570 (12h31m57.846707137s) [upload=12944]: Starting upload Aug 7 22:11:23.331668435 (12h31m57.846771012s) [upload=12944]: Done starting upload Aug 7 22:11:23.331728315 (12h31m57.846830881s) [upload=12944]: Uploading segments... Aug 7 22:11:23.331766773 (12h31m57.846869355s) [upload=12944]: Got next segment. Inline: true Aug 7 22:11:23.331771532 (12h31m57.846874108s) [upload=12944]: Waiting for error group managing segments... Aug 7 22:11:23.331812523 (12h31m57.846915099s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:23.331817222 (12h31m57.846919788s) [upload=12944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:23.331820650 (12h31m57.846923226s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:23.301325187 (12h32m57.816427773s) [upload=12944]: Done uploading segments... Aug 7 22:12:23.301350364 (12h32m57.816452947s) [upload=12944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:33.831933438 (12h33m8.34703602s) [upload=12965]: Starting upload Aug 7 22:12:33.832018368 (12h33m8.34712093s) [upload=12965]: Done starting upload Aug 7 22:12:33.832086961 (12h33m8.347189539s) [upload=12965]: Uploading segments... Aug 7 22:12:33.832135510 (12h33m8.347238093s) [upload=12965]: Got next segment. Inline: true Aug 7 22:12:33.832141249 (12h33m8.347243826s) [upload=12965]: Waiting for error group managing segments... Aug 7 22:12:33.832194093 (12h33m8.347296674s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:33.832199847 (12h33m8.347302426s) [upload=12965]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:33.832205088 (12h33m8.347307666s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:33.802839016 (12h34m8.317941593s) [upload=12965]: Done uploading segments... Aug 7 22:13:33.802913313 (12h34m8.31801589s) [upload=12965]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.913275619 (12h34m8.428378249s) [upload=12977]: Starting upload Aug 7 22:13:33.913398357 (12h34m8.428500941s) [upload=12977]: Done starting upload Aug 7 22:13:33.913496779 (12h34m8.428599348s) [upload=12977]: Uploading segments... Aug 7 22:13:33.913568818 (12h34m8.428671424s) [upload=12977]: Got next segment. Inline: true Aug 7 22:13:33.913576485 (12h34m8.428679072s) [upload=12977]: Waiting for error group managing segments... Aug 7 22:13:33.913646668 (12h34m8.428749234s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.913652247 (12h34m8.428754834s) [upload=12977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.913655814 (12h34m8.428758404s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.874545239 (12h35m8.389647805s) [upload=12977]: Done uploading segments... Aug 7 22:14:33.874575517 (12h35m8.389678082s) [upload=12977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:38.961368058 (12h35m13.476470627s) [upload=12995]: Starting upload Aug 7 22:14:38.961455294 (12h35m13.476557856s) [upload=12995]: Done starting upload Aug 7 22:14:38.961527233 (12h35m13.476629807s) [upload=12995]: Uploading segments... Aug 7 22:14:38.961574411 (12h35m13.476676997s) [upload=12995]: Got next segment. Inline: true Aug 7 22:14:38.961580546 (12h35m13.47668312s) [upload=12995]: Waiting for error group managing segments... Aug 7 22:14:38.961644878 (12h35m13.476747457s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:38.961650553 (12h35m13.476753133s) [upload=12995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:38.961655154 (12h35m13.476757734s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:38.926876179 (12h36m13.441978762s) [upload=12995]: Done uploading segments... Aug 7 22:15:38.926898754 (12h36m13.442001334s) [upload=12995]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000540277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XX.X..X..X..........................XXXXXX] duration: 30.000717192s FAIL (2025-08-07 22:15:37 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000735007s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XX...XXX...............................XXXXX] duration: 30.0016482s FAIL (2025-08-07 22:15:35 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 917334 (12h28m10.78401992s) [upload=12881]: Starting upload Aug 7 22:07:36.269068636 (12h28m10.784171213s) [upload=12881]: Done starting upload Aug 7 22:07:36.269153046 (12h28m10.784255626s) [upload=12881]: Uploading segments... Aug 7 22:07:36.269334617 (12h28m10.784437201s) [upload=12881]: Got next segment. Inline: true Aug 7 22:07:36.269343366 (12h28m10.784445949s) [upload=12881]: Waiting for error group managing segments... Aug 7 22:07:36.269420467 (12h28m10.784523048s) [upload=12881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:36.269425946 (12h28m10.784528526s) [upload=12881]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:36.269444491 (12h28m10.784547069s) [upload=12881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:36.519397428 (12h28m11.034500005s) [upload=12881]: Done uploading segments... Aug 7 22:07:36.519420980 (12h28m11.034523567s) [upload=12881]: Upload finished. err: <nil> Aug 7 22:08:28.430793526 (12h29m2.945896103s) [upload=12896]: Starting upload Aug 7 22:08:28.430857402 (12h29m2.94595997s) [upload=12896]: Done starting upload Aug 7 22:08:28.430919754 (12h29m2.946022323s) [upload=12896]: Uploading segments... Aug 7 22:08:28.430957561 (12h29m2.946060143s) [upload=12896]: Got next segment. Inline: true Aug 7 22:08:28.430962783 (12h29m2.946065362s) [upload=12896]: Waiting for error group managing segments... Aug 7 22:08:28.431004484 (12h29m2.946107061s) [upload=12896]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:28.431008904 (12h29m2.946111481s) [upload=12896]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:28.431012134 (12h29m2.946114712s) [upload=12896]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:28.505132712 (12h29m3.020235288s) [upload=12896]: Done uploading segments... Aug 7 22:08:28.505162495 (12h29m3.020265065s) [upload=12896]: Upload finished. err: <nil> Aug 7 22:09:09.304833839 (12h29m43.819936409s) [upload=12910]: Starting upload Aug 7 22:09:09.304915875 (12h29m43.820018452s) [upload=12910]: Done starting upload Aug 7 22:09:09.305006211 (12h29m43.820108779s) [upload=12910]: Uploading segments... Aug 7 22:09:09.305087545 (12h29m43.820190134s) [upload=12910]: Got next segment. Inline: true Aug 7 22:09:09.305095349 (12h29m43.820197936s) [upload=12910]: Waiting for error group managing segments... Aug 7 22:09:09.305151615 (12h29m43.820254177s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:09.305157403 (12h29m43.820259965s) [upload=12910]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:09.305160723 (12h29m43.820263285s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:09.641838505 (12h29m44.156941082s) [upload=12910]: Done uploading segments... Aug 7 22:09:09.641856719 (12h29m44.156959297s) [upload=12910]: Upload finished. err: <nil> Aug 7 22:09:35.598760483 (12h30m10.113863059s) [upload=12914]: Starting upload Aug 7 22:09:35.598830860 (12h30m10.113933439s) [upload=12914]: Done starting upload Aug 7 22:09:35.598909885 (12h30m10.11401245s) [upload=12914]: Uploading segments... Aug 7 22:09:35.598948468 (12h30m10.114051051s) [upload=12914]: Got next segment. Inline: true Aug 7 22:09:35.598952824 (12h30m10.114055403s) [upload=12914]: Waiting for error group managing segments... Aug 7 22:09:35.599006219 (12h30m10.114108795s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:35.599011436 (12h30m10.114114015s) [upload=12914]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:35.599014692 (12h30m10.114117272s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:35.668939111 (12h30m10.184041679s) [upload=12914]: Done uploading segments... Aug 7 22:09:35.668959039 (12h30m10.184061615s) [upload=12914]: Upload finished. err: <nil> Aug 7 22:10:18.242162305 (12h30m52.757264891s) [upload=12926]: Starting upload Aug 7 22:10:18.242275276 (12h30m52.75737786s) [upload=12926]: Done starting upload Aug 7 22:10:18.242410011 (12h30m52.757512577s) [upload=12926]: Uploading segments... Aug 7 22:10:18.242471376 (12h30m52.757573961s) [upload=12926]: Got next segment. Inline: true Aug 7 22:10:18.242478056 (12h30m52.757580635s) [upload=12926]: Waiting for error group managing segments... Aug 7 22:10:18.242535768 (12h30m52.757638347s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:18.242540806 (12h30m52.757643383s) [upload=12926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:18.242544109 (12h30m52.757646685s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:18.309606842 (12h30m52.824709422s) [upload=12926]: Done uploading segments... Aug 7 22:10:18.309625536 (12h30m52.824728119s) [upload=12926]: Upload finished. err: <nil> Aug 7 22:11:28.337393719 (12h32m2.852496289s) [upload=12946]: Starting upload Aug 7 22:11:28.337459886 (12h32m2.852562466s) [upload=12946]: Done starting upload Aug 7 22:11:28.337521996 (12h32m2.852624566s) [upload=12946]: Uploading segments... Aug 7 22:11:28.337560362 (12h32m2.852662948s) [upload=12946]: Got next segment. Inline: true Aug 7 22:11:28.337565059 (12h32m2.852667634s) [upload=12946]: Waiting for error group managing segments... Aug 7 22:11:28.337604389 (12h32m2.852706965s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:28.337608838 (12h32m2.852711414s) [upload=12946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:28.337612338 (12h32m2.852714915s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:58.307179904 (12h32m32.82228248s) [upload=12946]: Done uploading segments... Aug 7 22:11:58.307198958 (12h32m32.822301534s) [upload=12946]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:21.223007637 (12h32m55.738110218s) [upload=12959]: Starting upload Aug 7 22:12:21.223121202 (12h32m55.738223785s) [upload=12959]: Done starting upload Aug 7 22:12:21.223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001674159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 248.107322ms FAIL (2025-08-07 22:15:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 248.135485ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX....XX................................XXXXXX] duration: 30.002543991s FAIL (2025-08-07 22:15:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002587476s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 37.555582ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:15:15 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 37.577331ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXX.....X..........................XX] duration: 1m0.000949858s FAIL (2025-08-07 22:15:08 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== E36FN2Q, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.083671858 (12h31m52.598774435s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.083724464 (12h31m52.598827046s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084078846 (12h31m52.599181429s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084154873 (12h31m52.599257443s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084407796 (12h31m52.59951038s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084922562 (12h31m52.600025148s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084938723 (12h31m52.6000413s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085354116 (12h31m52.600456699s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085438941 (12h31m52.600541507s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085855643 (12h31m52.600958229s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085941726 (12h31m52.601044291s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.086348257 (12h31m52.601450834s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.086383562 (12h31m52.601486144s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.087350241 (12h31m52.602452811s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.088850336 (12h31m52.603952921s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.178629350 (12h31m52.693731915s) [upload=12943]: Next returned nil segment Aug 7 22:11:18.178647680 (12h31m52.693750267s) [upload=12943]: Waiting for error group managing segments... Aug 7 22:11:18.197176897 (12h31m52.712279474s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.201535099 (12h31m52.716637672s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.212513462 (12h31m52.727616038s) [noise=true , address=104.36.82.47:4552 , piece_id=NJTH3OJMQQBV6PEADF4IFYZ3U7QZHJIU774CEPVY5BBBXI6V53QA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.213955699 (12h31m52.729058282s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.214449276 (12h31m52.729551846s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.217895630 (12h31m52.732998217s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.218622391 (12h31m52.733724974s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.219619423 (12h31m52.734722004s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.227715032 (12h31m52.742817609s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.228516300 (12h31m52.743618876s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.241530379 (12h31m52.756632941s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.251935797 (12h31m52.767038375s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.277091280 (12h31m52.792193849s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291897936 (12h31m52.807000513s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291922152 (12h31m52.80702473s) [seg_pos={0 0}, upload=12943]: Segment reached optimal threshold of 14 pieces. Aug 7 22:11:18.292352474 (12h31m52.807455054s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:18.292749416 (12h31m52.807851996s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection Aug 7 22:11:18.293007428 (12h31m52.808110005s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: read tcp 159.223.143.39:34280->64.49.13.26:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293070269 (12h31m52.808172846s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: read tcp 159.223.143.39:46438->104.36.82.45:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293086568 (12h31m52.808189151s) [seg_pos={0 0}, upload=12943]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:11:18.293166574 (12h31m52.808269155s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:18.293172005 (12h31m52.808274588s) [upload=12943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:08.727892812 (12h34m43.242995378s) [upload=12943]: Done uploading segments... Aug 7 22:14:08.727919062 (12h34m43.243021627s) [upload=12943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:08.820786996 (12h34m43.335889564s) [upload=12987]: Starting upload Aug 7 22:14:08.820869466 (12h34m43.335972332s) [upload=12987]: Done starting upload Aug 7 22:14:08.820990812 (12h34m43.336093389s) [upload=12987]: Uploading segments... Aug 7 22:14:08.821180945 (12h34m43.336283523s) [upload=12987]: Got next segment. Inline: false Aug 7 22:14:08.821227735 (12h34m43.336330315s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:08.821233790 (12h34m43.336336478s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:08.788378642 (12h35m43.303481218s) [upload=12987]: Next segment err: context deadline exceeded Aug 7 22:15:08.788567212 (12h35m43.303669789s) [upload=12987]: Done uploading segments... Aug 7 22:15:08.788593806 (12h35m43.303696388s) [upload=12987]: Upload finished. err: context deadline exceeded Duration: 1m0.000966855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XX.X..X..X..........................XXXXX] duration: 30.000565227s FAIL (2025-08-07 22:15:06 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000587507s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.X....X.X.................................XXXXX] duration: 30.001516544s FAIL (2025-08-07 22:15:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00154063s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........................................X...] duration: 1.141575634s FAIL (2025-08-07 22:14:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 1.141609102s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XX...X.............................XXXX] duration: 1m0.001096078s FAIL (2025-08-07 22:14:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001119978s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 30.070705ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:14:51 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 30.093439ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX....XX................................XXXXX] duration: 30.001296192s FAIL (2025-08-07 22:14:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001311455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 308.821168ms FAIL (2025-08-07 22:14:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 308.846408ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XX...XXX...............................XXXX] duration: 30.001145005s FAIL (2025-08-07 22:14:34 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== r: <nil> Aug 7 22:07:10.243253097 (12h27m44.758355666s) [upload=12876]: Starting upload Aug 7 22:07:10.243304082 (12h27m44.758406651s) [upload=12876]: Done starting upload Aug 7 22:07:10.243357543 (12h27m44.758460108s) [upload=12876]: Uploading segments... Aug 7 22:07:10.243394039 (12h27m44.758496624s) [upload=12876]: Got next segment. Inline: true Aug 7 22:07:10.243398567 (12h27m44.758501143s) [upload=12876]: Waiting for error group managing segments... Aug 7 22:07:10.243438205 (12h27m44.758540782s) [upload=12876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:10.243442395 (12h27m44.758544972s) [upload=12876]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:10.243445590 (12h27m44.758548166s) [upload=12876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:10.479504888 (12h27m44.994607467s) [upload=12876]: Done uploading segments... Aug 7 22:07:10.479529786 (12h27m44.994632364s) [upload=12876]: Upload finished. err: <nil> Aug 7 22:07:36.268917334 (12h28m10.78401992s) [upload=12881]: Starting upload Aug 7 22:07:36.269068636 (12h28m10.784171213s) [upload=12881]: Done starting upload Aug 7 22:07:36.269153046 (12h28m10.784255626s) [upload=12881]: Uploading segments... Aug 7 22:07:36.269334617 (12h28m10.784437201s) [upload=12881]: Got next segment. Inline: true Aug 7 22:07:36.269343366 (12h28m10.784445949s) [upload=12881]: Waiting for error group managing segments... Aug 7 22:07:36.269420467 (12h28m10.784523048s) [upload=12881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:36.269425946 (12h28m10.784528526s) [upload=12881]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:36.269444491 (12h28m10.784547069s) [upload=12881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:36.519397428 (12h28m11.034500005s) [upload=12881]: Done uploading segments... Aug 7 22:07:36.519420980 (12h28m11.034523567s) [upload=12881]: Upload finished. err: <nil> Aug 7 22:08:28.430793526 (12h29m2.945896103s) [upload=12896]: Starting upload Aug 7 22:08:28.430857402 (12h29m2.94595997s) [upload=12896]: Done starting upload Aug 7 22:08:28.430919754 (12h29m2.946022323s) [upload=12896]: Uploading segments... Aug 7 22:08:28.430957561 (12h29m2.946060143s) [upload=12896]: Got next segment. Inline: true Aug 7 22:08:28.430962783 (12h29m2.946065362s) [upload=12896]: Waiting for error group managing segments... Aug 7 22:08:28.431004484 (12h29m2.946107061s) [upload=12896]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:28.431008904 (12h29m2.946111481s) [upload=12896]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:28.431012134 (12h29m2.946114712s) [upload=12896]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:28.505132712 (12h29m3.020235288s) [upload=12896]: Done uploading segments... Aug 7 22:08:28.505162495 (12h29m3.020265065s) [upload=12896]: Upload finished. err: <nil> Aug 7 22:09:09.304833839 (12h29m43.819936409s) [upload=12910]: Starting upload Aug 7 22:09:09.304915875 (12h29m43.820018452s) [upload=12910]: Done starting upload Aug 7 22:09:09.305006211 (12h29m43.820108779s) [upload=12910]: Uploading segments... Aug 7 22:09:09.305087545 (12h29m43.820190134s) [upload=12910]: Got next segment. Inline: true Aug 7 22:09:09.305095349 (12h29m43.820197936s) [upload=12910]: Waiting for error group managing segments... Aug 7 22:09:09.305151615 (12h29m43.820254177s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:09.305157403 (12h29m43.820259965s) [upload=12910]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:09.305160723 (12h29m43.820263285s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:09.641838505 (12h29m44.156941082s) [upload=12910]: Done uploading segments... Aug 7 22:09:09.641856719 (12h29m44.156959297s) [upload=12910]: Upload finished. err: <nil> Aug 7 22:09:35.598760483 (12h30m10.113863059s) [upload=12914]: Starting upload Aug 7 22:09:35.598830860 (12h30m10.113933439s) [upload=12914]: Done starting upload Aug 7 22:09:35.598909885 (12h30m10.11401245s) [upload=12914]: Uploading segments... Aug 7 22:09:35.598948468 (12h30m10.114051051s) [upload=12914]: Got next segment. Inline: true Aug 7 22:09:35.598952824 (12h30m10.114055403s) [upload=12914]: Waiting for error group managing segments... Aug 7 22:09:35.599006219 (12h30m10.114108795s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:35.599011436 (12h30m10.114114015s) [upload=12914]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:35.599014692 (12h30m10.114117272s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:35.668939111 (12h30m10.184041679s) [upload=12914]: Done uploading segments... Aug 7 22:09:35.668959039 (12h30m10.184061615s) [upload=12914]: Upload finished. err: <nil> Aug 7 22:10:18.242162305 (12h30m52.757264891s) [upload=12926]: Starting upload Aug 7 22:10:18.242275276 (12h30m52.75737786s) [upload=12926]: Done starting upload Aug 7 22:10:18.242410011 (12h30m52.757512577s) [upload=12926]: Uploading segments... Aug 7 22:10:18.242471376 (12h30m52.757573961s) [upload=12926]: Got next segment. Inline: true Aug 7 22:10:18.242478056 (12h30m52.757580635s) [upload=12926]: Waiting for error group managing segments... Aug 7 22:10:18.242535768 (12h30m52.757638347s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:18.242540806 (12h30m52.757643383s) [upload=12926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:18.242544109 (12h30m52.757646685s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:18.309606842 (12h30m52.824709422s) [upload=12926]: Done uploading segments... Aug 7 22:10:18.309625536 (12h30m52.824728119s) [upload=12926]: Upload finished. err: <nil> Aug 7 22:11:28.337393719 (12h32m2.852496289s) [upload=12946]: Starting upload Aug 7 22:11:28.337459886 (12h32m2.852562466s) [upload=12946]: Done starting upload Aug 7 22:11:28.337521996 (12h32m2.852624566s) [upload=12946]: Uploading segments... Aug 7 22:11:28.337560362 (12h32m2.852662948s) [upload=12946]: Got next segment. Inline: true Aug 7 22:11:28.337565059 (12h32m2.852667634s) [upload=12946]: Waiting for error group managing segments... Aug 7 22:11:28.337604389 (12h32m2.852706965s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:28.337608838 (12h32m2.852711414s) [upload=12946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:28.337612338 (12h32m2.852714915s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:58.307179904 (12h32m32.82228248s) [upload=12946]: Done uploading segments... Aug 7 22:11:58.307198958 (12h32m32.822301534s) [upload=12946]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:21.223007637 (12h32m55.738110218s) [upload=12959]: Starting upload Aug 7 22:12:21.223121202 (12h32m55.738223785s) [upload=12959]: Done starting upload Aug 7 22:12:21.223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001170434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.X.................................XXX] duration: 1m0.000775492s FAIL (2025-08-07 22:14:33 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 529011s) [upload=12845]: Upload finished. err: <nil> Aug 7 22:06:04.189782266 (12h26m38.704884848s) [upload=12860]: Starting upload Aug 7 22:06:04.189892755 (12h26m38.704995341s) [upload=12860]: Done starting upload Aug 7 22:06:04.190023263 (12h26m38.705125829s) [upload=12860]: Uploading segments... Aug 7 22:06:04.190144117 (12h26m38.705246697s) [upload=12860]: Got next segment. Inline: true Aug 7 22:06:04.190151584 (12h26m38.705254162s) [upload=12860]: Waiting for error group managing segments... Aug 7 22:06:04.190204381 (12h26m38.705306957s) [upload=12860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:04.190216933 (12h26m38.70531951s) [upload=12860]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:04.190221696 (12h26m38.705324273s) [upload=12860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:04.253115591 (12h26m38.768218174s) [upload=12860]: Done uploading segments... Aug 7 22:06:04.253153336 (12h26m38.768255914s) [upload=12860]: Upload finished. err: <nil> Aug 7 22:06:54.132475466 (12h27m28.647578034s) [upload=12873]: Starting upload Aug 7 22:06:54.132547886 (12h27m28.647650462s) [upload=12873]: Done starting upload Aug 7 22:06:54.132786697 (12h27m28.647889262s) [upload=12873]: Uploading segments... Aug 7 22:06:54.132870099 (12h27m28.647972686s) [upload=12873]: Got next segment. Inline: true Aug 7 22:06:54.132877110 (12h27m28.647979693s) [upload=12873]: Waiting for error group managing segments... Aug 7 22:06:54.132948498 (12h27m28.648051076s) [upload=12873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:54.132953672 (12h27m28.648056256s) [upload=12873]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:54.132956651 (12h27m28.648059229s) [upload=12873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:54.200614549 (12h27m28.71571713s) [upload=12873]: Done uploading segments... Aug 7 22:06:54.200630357 (12h27m28.715732933s) [upload=12873]: Upload finished. err: <nil> Aug 7 22:08:03.216043710 (12h28m37.731146297s) [upload=12889]: Starting upload Aug 7 22:08:03.216136680 (12h28m37.731239259s) [upload=12889]: Done starting upload Aug 7 22:08:03.216283833 (12h28m37.731386404s) [upload=12889]: Uploading segments... Aug 7 22:08:03.216375553 (12h28m37.73147814s) [upload=12889]: Got next segment. Inline: true Aug 7 22:08:03.216382176 (12h28m37.731484755s) [upload=12889]: Waiting for error group managing segments... Aug 7 22:08:03.216448592 (12h28m37.73155116s) [upload=12889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:03.216454041 (12h28m37.731556619s) [upload=12889]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:03.216457480 (12h28m37.731560058s) [upload=12889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:03.293854136 (12h28m37.808956726s) [upload=12889]: Done uploading segments... Aug 7 22:08:03.293877099 (12h28m37.80897968s) [upload=12889]: Upload finished. err: <nil> Aug 7 22:08:26.591681271 (12h29m1.106783851s) [upload=12895]: Starting upload Aug 7 22:08:26.591770003 (12h29m1.106872567s) [upload=12895]: Done starting upload Aug 7 22:08:26.591871625 (12h29m1.106974202s) [upload=12895]: Uploading segments... Aug 7 22:08:26.591931014 (12h29m1.107033599s) [upload=12895]: Got next segment. Inline: true Aug 7 22:08:26.591936202 (12h29m1.107038768s) [upload=12895]: Waiting for error group managing segments... Aug 7 22:08:26.591990877 (12h29m1.107093457s) [upload=12895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:26.591995259 (12h29m1.107097828s) [upload=12895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:26.591998403 (12h29m1.107100969s) [upload=12895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:26.672830843 (12h29m1.187933663s) [upload=12895]: Done uploading segments... Aug 7 22:08:26.672861348 (12h29m1.187963926s) [upload=12895]: Upload finished. err: <nil> Aug 7 22:08:55.672477854 (12h29m30.187580437s) [upload=12903]: Starting upload Aug 7 22:08:55.672581371 (12h29m30.187684016s) [upload=12903]: Done starting upload Aug 7 22:08:55.672796946 (12h29m30.187899524s) [upload=12903]: Uploading segments... Aug 7 22:08:55.672988832 (12h29m30.188091398s) [upload=12903]: Got next segment. Inline: true Aug 7 22:08:55.673020300 (12h29m30.188122877s) [upload=12903]: Waiting for error group managing segments... Aug 7 22:08:55.673111302 (12h29m30.188213889s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:55.673117396 (12h29m30.188219973s) [upload=12903]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:55.673121451 (12h29m30.188224017s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:55.748990886 (12h29m30.264093475s) [upload=12903]: Done uploading segments... Aug 7 22:08:55.749011264 (12h29m30.264113844s) [upload=12903]: Upload finished. err: <nil> Aug 7 22:09:33.459458247 (12h30m7.974560823s) [upload=12913]: Starting upload Aug 7 22:09:33.459560333 (12h30m7.974662911s) [upload=12913]: Done starting upload Aug 7 22:09:33.459666234 (12h30m7.974768811s) [upload=12913]: Uploading segments... Aug 7 22:09:33.459731513 (12h30m7.974834099s) [upload=12913]: Got next segment. Inline: true Aug 7 22:09:33.459737908 (12h30m7.974840491s) [upload=12913]: Waiting for error group managing segments... Aug 7 22:09:33.459810629 (12h30m7.974913195s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:33.459816521 (12h30m7.974919097s) [upload=12913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:33.459819752 (12h30m7.974922328s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:33.522052292 (12h30m8.037154862s) [upload=12913]: Done uploading segments... Aug 7 22:09:33.522072074 (12h30m8.037174651s) [upload=12913]: Upload finished. err: <nil> Aug 7 22:10:20.772089130 (12h30m55.287191696s) [upload=12928]: Starting upload Aug 7 22:10:20.772161634 (12h30m55.28726421s) [upload=12928]: Done starting upload Aug 7 22:10:20.772231905 (12h30m55.287334472s) [upload=12928]: Uploading segments... Aug 7 22:10:20.772284367 (12h30m55.287386947s) [upload=12928]: Got next segment. Inline: true Aug 7 22:10:20.772289092 (12h30m55.287391658s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:10:20.772344988 (12h30m55.287447554s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:20.772349640 (12h30m55.287452217s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:20.772352965 (12h30m55.287455541s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:20.857003207 (12h30m55.372105787s) [upload=12928]: Done uploading segments... Aug 7 22:10:20.857032212 (12h30m55.372134783s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:11:23.331604570 (12h31m57.846707137s) [upload=12944]: Starting upload Aug 7 22:11:23.331668435 (12h31m57.846771012s) [upload=12944]: Done starting upload Aug 7 22:11:23.331728315 (12h31m57.846830881s) [upload=12944]: Uploading segments... Aug 7 22:11:23.331766773 (12h31m57.846869355s) [upload=12944]: Got next segment. Inline: true Aug 7 22:11:23.331771532 (12h31m57.846874108s) [upload=12944]: Waiting for error group managing segments... Aug 7 22:11:23.331812523 (12h31m57.846915099s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:23.331817222 (12h31m57.846919788s) [upload=12944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:23.331820650 (12h31m57.846923226s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:23.301325187 (12h32m57.816427773s) [upload=12944]: Done uploading segments... Aug 7 22:12:23.301350364 (12h32m57.816452947s) [upload=12944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:33.831933438 (12h33m8.34703602s) [upload=12965]: Starting upload Aug 7 22:12:33.832018368 (12h33m8.34712093s) [upload=12965]: Done starting upload Aug 7 22:12:33.832086961 (12h33m8.347189539s) [upload=12965]: Uploading segments... Aug 7 22:12:33.832135510 (12h33m8.347238093s) [upload=12965]: Got next segment. Inline: true Aug 7 22:12:33.832141249 (12h33m8.347243826s) [upload=12965]: Waiting for error group managing segments... Aug 7 22:12:33.832194093 (12h33m8.347296674s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:33.832199847 (12h33m8.347302426s) [upload=12965]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:33.832205088 (12h33m8.347307666s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:33.802839016 (12h34m8.317941593s) [upload=12965]: Done uploading segments... Aug 7 22:13:33.802913313 (12h34m8.31801589s) [upload=12965]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.913275619 (12h34m8.428378249s) [upload=12977]: Starting upload Aug 7 22:13:33.913398357 (12h34m8.428500941s) [upload=12977]: Done starting upload Aug 7 22:13:33.913496779 (12h34m8.428599348s) [upload=12977]: Uploading segments... Aug 7 22:13:33.913568818 (12h34m8.428671424s) [upload=12977]: Got next segment. Inline: true Aug 7 22:13:33.913576485 (12h34m8.428679072s) [upload=12977]: Waiting for error group managing segments... Aug 7 22:13:33.913646668 (12h34m8.428749234s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.913652247 (12h34m8.428754834s) [upload=12977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.913655814 (12h34m8.428758404s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.874545239 (12h35m8.389647805s) [upload=12977]: Done uploading segments... Aug 7 22:14:33.874575517 (12h35m8.389678082s) [upload=12977]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000789031s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 34.249291ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:14:31 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 34.291851ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 235.934613ms FAIL (2025-08-07 22:14:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 235.956363ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X....X.X.................................XXXX] duration: 30.000940893s FAIL (2025-08-07 22:14:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000962339s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XX.X..X..X..........................XXXX] duration: 30.009237784s FAIL (2025-08-07 22:14:27 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.009267304s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [......XX..XX................................................] duration: 1m8.218902545s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:14:19 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 1m8.218945949s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX....XX................................XXXX] duration: 30.00089923s FAIL (2025-08-07 22:14:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000923931s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXX.....X..........................X] duration: 2m51.234772261s FAIL (2025-08-07 22:14:08 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== tBegin Aug 7 22:11:17.525359064 (12h31m52.040461628s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:11:18.082447170 (12h31m52.597549737s) [seg_pos={0 0}, upload=12943]: Begin upload segment... Aug 7 22:11:18.082514390 (12h31m52.597616967s) [seg_pos={0 0}, upload=12943]: Done begin upload segment. Aug 7 22:11:18.082538608 (12h31m52.597641173s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.083154454 (12h31m52.598257021s) [noise=true , address=104.36.82.47:4552 , piece_id=NJTH3OJMQQBV6PEADF4IFYZ3U7QZHJIU774CEPVY5BBBXI6V53QA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.083161654 (12h31m52.598264238s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.083671858 (12h31m52.598774435s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.083724464 (12h31m52.598827046s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084078846 (12h31m52.599181429s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084154873 (12h31m52.599257443s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084407796 (12h31m52.59951038s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084922562 (12h31m52.600025148s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.084938723 (12h31m52.6000413s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085354116 (12h31m52.600456699s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085438941 (12h31m52.600541507s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085855643 (12h31m52.600958229s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.085941726 (12h31m52.601044291s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.086348257 (12h31m52.601450834s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.086383562 (12h31m52.601486144s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.087350241 (12h31m52.602452811s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.088850336 (12h31m52.603952921s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.178629350 (12h31m52.693731915s) [upload=12943]: Next returned nil segment Aug 7 22:11:18.178647680 (12h31m52.693750267s) [upload=12943]: Waiting for error group managing segments... Aug 7 22:11:18.197176897 (12h31m52.712279474s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.201535099 (12h31m52.716637672s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.212513462 (12h31m52.727616038s) [noise=true , address=104.36.82.47:4552 , piece_id=NJTH3OJMQQBV6PEADF4IFYZ3U7QZHJIU774CEPVY5BBBXI6V53QA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.213955699 (12h31m52.729058282s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.214449276 (12h31m52.729551846s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.217895630 (12h31m52.732998217s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.218622391 (12h31m52.733724974s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.219619423 (12h31m52.734722004s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.227715032 (12h31m52.742817609s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.228516300 (12h31m52.743618876s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.241530379 (12h31m52.756632941s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.251935797 (12h31m52.767038375s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.277091280 (12h31m52.792193849s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291897936 (12h31m52.807000513s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291922152 (12h31m52.80702473s) [seg_pos={0 0}, upload=12943]: Segment reached optimal threshold of 14 pieces. Aug 7 22:11:18.292352474 (12h31m52.807455054s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:18.292749416 (12h31m52.807851996s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection Aug 7 22:11:18.293007428 (12h31m52.808110005s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: read tcp 159.223.143.39:34280->64.49.13.26:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293070269 (12h31m52.808172846s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: read tcp 159.223.143.39:46438->104.36.82.45:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293086568 (12h31m52.808189151s) [seg_pos={0 0}, upload=12943]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:11:18.293166574 (12h31m52.808269155s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:18.293172005 (12h31m52.808274588s) [upload=12943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:08.727892812 (12h34m43.242995378s) [upload=12943]: Done uploading segments... Aug 7 22:14:08.727919062 (12h34m43.243021627s) [upload=12943]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2m51.234867056s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 29.589979ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:14:07 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 29.617733ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XX...XXX...............................XXX] duration: 30.001103287s FAIL (2025-08-07 22:14:03 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9s) [upload=12858]: Upload finished. err: <nil> Aug 7 22:06:32.148888159 (12h27m6.663990738s) [upload=12865]: Starting upload Aug 7 22:06:32.149072990 (12h27m6.664175568s) [upload=12865]: Done starting upload Aug 7 22:06:32.149113469 (12h27m6.664216039s) [upload=12865]: Uploading segments... Aug 7 22:06:32.149198941 (12h27m6.664301523s) [upload=12865]: Got next segment. Inline: true Aug 7 22:06:32.149204391 (12h27m6.664306967s) [upload=12865]: Waiting for error group managing segments... Aug 7 22:06:32.149254207 (12h27m6.664356784s) [upload=12865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:32.149259883 (12h27m6.664362461s) [upload=12865]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:32.149263154 (12h27m6.664365731s) [upload=12865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:32.215291954 (12h27m6.730394521s) [upload=12865]: Done uploading segments... Aug 7 22:06:32.215328701 (12h27m6.730431277s) [upload=12865]: Upload finished. err: <nil> Aug 7 22:07:10.243253097 (12h27m44.758355666s) [upload=12876]: Starting upload Aug 7 22:07:10.243304082 (12h27m44.758406651s) [upload=12876]: Done starting upload Aug 7 22:07:10.243357543 (12h27m44.758460108s) [upload=12876]: Uploading segments... Aug 7 22:07:10.243394039 (12h27m44.758496624s) [upload=12876]: Got next segment. Inline: true Aug 7 22:07:10.243398567 (12h27m44.758501143s) [upload=12876]: Waiting for error group managing segments... Aug 7 22:07:10.243438205 (12h27m44.758540782s) [upload=12876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:10.243442395 (12h27m44.758544972s) [upload=12876]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:10.243445590 (12h27m44.758548166s) [upload=12876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:10.479504888 (12h27m44.994607467s) [upload=12876]: Done uploading segments... Aug 7 22:07:10.479529786 (12h27m44.994632364s) [upload=12876]: Upload finished. err: <nil> Aug 7 22:07:36.268917334 (12h28m10.78401992s) [upload=12881]: Starting upload Aug 7 22:07:36.269068636 (12h28m10.784171213s) [upload=12881]: Done starting upload Aug 7 22:07:36.269153046 (12h28m10.784255626s) [upload=12881]: Uploading segments... Aug 7 22:07:36.269334617 (12h28m10.784437201s) [upload=12881]: Got next segment. Inline: true Aug 7 22:07:36.269343366 (12h28m10.784445949s) [upload=12881]: Waiting for error group managing segments... Aug 7 22:07:36.269420467 (12h28m10.784523048s) [upload=12881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:36.269425946 (12h28m10.784528526s) [upload=12881]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:36.269444491 (12h28m10.784547069s) [upload=12881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:36.519397428 (12h28m11.034500005s) [upload=12881]: Done uploading segments... Aug 7 22:07:36.519420980 (12h28m11.034523567s) [upload=12881]: Upload finished. err: <nil> Aug 7 22:08:28.430793526 (12h29m2.945896103s) [upload=12896]: Starting upload Aug 7 22:08:28.430857402 (12h29m2.94595997s) [upload=12896]: Done starting upload Aug 7 22:08:28.430919754 (12h29m2.946022323s) [upload=12896]: Uploading segments... Aug 7 22:08:28.430957561 (12h29m2.946060143s) [upload=12896]: Got next segment. Inline: true Aug 7 22:08:28.430962783 (12h29m2.946065362s) [upload=12896]: Waiting for error group managing segments... Aug 7 22:08:28.431004484 (12h29m2.946107061s) [upload=12896]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:28.431008904 (12h29m2.946111481s) [upload=12896]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:28.431012134 (12h29m2.946114712s) [upload=12896]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:28.505132712 (12h29m3.020235288s) [upload=12896]: Done uploading segments... Aug 7 22:08:28.505162495 (12h29m3.020265065s) [upload=12896]: Upload finished. err: <nil> Aug 7 22:09:09.304833839 (12h29m43.819936409s) [upload=12910]: Starting upload Aug 7 22:09:09.304915875 (12h29m43.820018452s) [upload=12910]: Done starting upload Aug 7 22:09:09.305006211 (12h29m43.820108779s) [upload=12910]: Uploading segments... Aug 7 22:09:09.305087545 (12h29m43.820190134s) [upload=12910]: Got next segment. Inline: true Aug 7 22:09:09.305095349 (12h29m43.820197936s) [upload=12910]: Waiting for error group managing segments... Aug 7 22:09:09.305151615 (12h29m43.820254177s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:09.305157403 (12h29m43.820259965s) [upload=12910]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:09.305160723 (12h29m43.820263285s) [upload=12910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:09.641838505 (12h29m44.156941082s) [upload=12910]: Done uploading segments... Aug 7 22:09:09.641856719 (12h29m44.156959297s) [upload=12910]: Upload finished. err: <nil> Aug 7 22:09:35.598760483 (12h30m10.113863059s) [upload=12914]: Starting upload Aug 7 22:09:35.598830860 (12h30m10.113933439s) [upload=12914]: Done starting upload Aug 7 22:09:35.598909885 (12h30m10.11401245s) [upload=12914]: Uploading segments... Aug 7 22:09:35.598948468 (12h30m10.114051051s) [upload=12914]: Got next segment. Inline: true Aug 7 22:09:35.598952824 (12h30m10.114055403s) [upload=12914]: Waiting for error group managing segments... Aug 7 22:09:35.599006219 (12h30m10.114108795s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:35.599011436 (12h30m10.114114015s) [upload=12914]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:35.599014692 (12h30m10.114117272s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:35.668939111 (12h30m10.184041679s) [upload=12914]: Done uploading segments... Aug 7 22:09:35.668959039 (12h30m10.184061615s) [upload=12914]: Upload finished. err: <nil> Aug 7 22:10:18.242162305 (12h30m52.757264891s) [upload=12926]: Starting upload Aug 7 22:10:18.242275276 (12h30m52.75737786s) [upload=12926]: Done starting upload Aug 7 22:10:18.242410011 (12h30m52.757512577s) [upload=12926]: Uploading segments... Aug 7 22:10:18.242471376 (12h30m52.757573961s) [upload=12926]: Got next segment. Inline: true Aug 7 22:10:18.242478056 (12h30m52.757580635s) [upload=12926]: Waiting for error group managing segments... Aug 7 22:10:18.242535768 (12h30m52.757638347s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:18.242540806 (12h30m52.757643383s) [upload=12926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:18.242544109 (12h30m52.757646685s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:18.309606842 (12h30m52.824709422s) [upload=12926]: Done uploading segments... Aug 7 22:10:18.309625536 (12h30m52.824728119s) [upload=12926]: Upload finished. err: <nil> Aug 7 22:11:28.337393719 (12h32m2.852496289s) [upload=12946]: Starting upload Aug 7 22:11:28.337459886 (12h32m2.852562466s) [upload=12946]: Done starting upload Aug 7 22:11:28.337521996 (12h32m2.852624566s) [upload=12946]: Uploading segments... Aug 7 22:11:28.337560362 (12h32m2.852662948s) [upload=12946]: Got next segment. Inline: true Aug 7 22:11:28.337565059 (12h32m2.852667634s) [upload=12946]: Waiting for error group managing segments... Aug 7 22:11:28.337604389 (12h32m2.852706965s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:28.337608838 (12h32m2.852711414s) [upload=12946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:28.337612338 (12h32m2.852714915s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:58.307179904 (12h32m32.82228248s) [upload=12946]: Done uploading segments... Aug 7 22:11:58.307198958 (12h32m32.822301534s) [upload=12946]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:21.223007637 (12h32m55.738110218s) [upload=12959]: Starting upload Aug 7 22:12:21.223121202 (12h32m55.738223785s) [upload=12959]: Done starting upload Aug 7 22:12:21.223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001129748s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)