Orbiter History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.XX..XX................................................X..XX] duration: 2m0.001413175s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:22:29 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.001466367s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XX...X.............................XXXXXXXXXXX] duration: 1m0.001836632s FAIL (2025-08-07 22:22:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001853513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXX.XX.X..X.............................XXX] duration: 30.001348811s FAIL (2025-08-07 22:22:06 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001384342s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXX.XX...XXX...............................XXXXXXXXXXXXXX] duration: 30.001531009s FAIL (2025-08-07 22:21:59 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:19.727995115 (12h36m54.243097692s) [upload=13016]: Starting upload Aug 7 22:16:19.728064428 (12h36m54.243167004s) [upload=13016]: Done starting upload Aug 7 22:16:19.728134488 (12h36m54.243237051s) [upload=13016]: Uploading segments... Aug 7 22:16:19.728182154 (12h36m54.243284736s) [upload=13016]: Got next segment. Inline: true Aug 7 22:16:19.728186380 (12h36m54.243288955s) [upload=13016]: Waiting for error group managing segments... Aug 7 22:16:19.728223320 (12h36m54.243325899s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:19.728227150 (12h36m54.243329727s) [upload=13016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:19.728229527 (12h36m54.243332103s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:49.691229995 (12h37m24.206332575s) [upload=13016]: Done uploading segments... Aug 7 22:16:49.691252411 (12h37m24.206354988s) [upload=13016]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:13.655336229 (12h37m48.170438797s) [upload=13028]: Starting upload Aug 7 22:17:13.655409916 (12h37m48.170512492s) [upload=13028]: Done starting upload Aug 7 22:17:13.655478262 (12h37m48.17058083s) [upload=13028]: Uploading segments... Aug 7 22:17:13.655548537 (12h37m48.17065113s) [upload=13028]: Got next segment. Inline: true Aug 7 22:17:13.655555256 (12h37m48.170657826s) [upload=13028]: Waiting for error group managing segments... Aug 7 22:17:13.655620901 (12h37m48.170723469s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:13.655627928 (12h37m48.170730508s) [upload=13028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:13.655632202 (12h37m48.170734786s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:43.624980277 (12h38m18.140082854s) [upload=13028]: Done uploading segments... Aug 7 22:17:43.624997725 (12h38m18.140100301s) [upload=13028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:43.747114121 (12h38m18.2622167s) [upload=13037]: Starting upload Aug 7 22:17:43.747186113 (12h38m18.262288692s) [upload=13037]: Done starting upload Aug 7 22:17:43.747260024 (12h38m18.262362591s) [upload=13037]: Uploading segments... Aug 7 22:17:43.747409410 (12h38m18.262511987s) [upload=13037]: Got next segment. Inline: true Aug 7 22:17:43.747414935 (12h38m18.262517512s) [upload=13037]: Waiting for error group managing segments... Aug 7 22:17:43.747457457 (12h38m18.262560036s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:43.747461997 (12h38m18.262564572s) [upload=13037]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:43.747465159 (12h38m18.262567736s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:13.715508579 (12h38m48.230611159s) [upload=13037]: Done uploading segments... Aug 7 22:18:13.715531957 (12h38m48.230634536s) [upload=13037]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:33.297944141 (12h39m7.813046721s) [upload=13046]: Starting upload Aug 7 22:18:33.298041745 (12h39m7.813144331s) [upload=13046]: Done starting upload Aug 7 22:18:33.298133963 (12h39m7.813236527s) [upload=13046]: Uploading segments... Aug 7 22:18:33.298195648 (12h39m7.813298236s) [upload=13046]: Got next segment. Inline: true Aug 7 22:18:33.298201348 (12h39m7.813303934s) [upload=13046]: Waiting for error group managing segments... Aug 7 22:18:33.298250060 (12h39m7.813352642s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:33.298255690 (12h39m7.813358273s) [upload=13046]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:33.298259809 (12h39m7.813362387s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:03.266409729 (12h39m37.781512306s) [upload=13046]: Done uploading segments... Aug 7 22:19:03.266458971 (12h39m37.781561549s) [upload=13046]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:30.020553557 (12h40m4.535656125s) [upload=13060]: Starting upload Aug 7 22:19:30.020630355 (12h40m4.535732931s) [upload=13060]: Done starting upload Aug 7 22:19:30.020870620 (12h40m4.535973198s) [upload=13060]: Uploading segments... Aug 7 22:19:30.020984540 (12h40m4.536087103s) [upload=13060]: Got next segment. Inline: true Aug 7 22:19:30.020990929 (12h40m4.536093523s) [upload=13060]: Waiting for error group managing segments... Aug 7 22:19:30.021057882 (12h40m4.536160461s) [upload=13060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:30.021063707 (12h40m4.536166287s) [upload=13060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:30.021067070 (12h40m4.536169633s) [upload=13060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:59.990140275 (12h40m34.505242847s) [upload=13060]: Done uploading segments... Aug 7 22:19:59.990166176 (12h40m34.505268763s) [upload=13060]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:14.462825235 (12h40m48.977927811s) [upload=13071]: Starting upload Aug 7 22:20:14.462946370 (12h40m48.978048965s) [upload=13071]: Done starting upload Aug 7 22:20:14.463094242 (12h40m48.978196809s) [upload=13071]: Uploading segments... Aug 7 22:20:14.463162579 (12h40m48.978265164s) [upload=13071]: Got next segment. Inline: true Aug 7 22:20:14.463168150 (12h40m48.978270729s) [upload=13071]: Waiting for error group managing segments... Aug 7 22:20:14.463216881 (12h40m48.978319457s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:14.463226562 (12h40m48.978329138s) [upload=13071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:14.463230088 (12h40m48.978332654s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:44.419309349 (12h41m18.934411917s) [upload=13071]: Done uploading segments... Aug 7 22:20:44.419327995 (12h41m18.934430565s) [upload=13071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:59.026227228 (12h41m33.541329811s) [upload=13081]: Starting upload Aug 7 22:20:59.026335834 (12h41m33.541438465s) [upload=13081]: Done starting upload Aug 7 22:20:59.026413148 (12h41m33.541515725s) [upload=13081]: Uploading segments... Aug 7 22:20:59.026487957 (12h41m33.541590544s) [upload=13081]: Got next segment. Inline: true Aug 7 22:20:59.026494660 (12h41m33.54159724s) [upload=13081]: Waiting for error group managing segments... Aug 7 22:20:59.026548283 (12h41m33.541650867s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:59.026554014 (12h41m33.541656591s) [upload=13081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:59.026557057 (12h41m33.541659621s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:28.995870666 (12h42m3.510973249s) [upload=13081]: Done uploading segments... Aug 7 22:21:28.995890051 (12h42m3.510992681s) [upload=13081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:29.129769938 (12h42m3.644872515s) [upload=13090]: Starting upload Aug 7 22:21:29.129856429 (12h42m3.644959013s) [upload=13090]: Done starting upload Aug 7 22:21:29.129943698 (12h42m3.645046276s) [upload=13090]: Uploading segments... Aug 7 22:21:29.129985259 (12h42m3.645087845s) [upload=13090]: Got next segment. Inline: true Aug 7 22:21:29.129990476 (12h42m3.645093052s) [upload=13090]: Waiting for error group managing segments... Aug 7 22:21:29.130027566 (12h42m3.645130145s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:29.130031892 (12h42m3.645134469s) [upload=13090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:29.130035507 (12h42m3.645138082s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:59.086555047 (12h42m33.601657631s) [upload=13090]: Done uploading segments... Aug 7 22:21:59.086578788 (12h42m33.601681377s) [upload=13090]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001549588s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXX....XX................................XXXXXXXXXXXXXXX] duration: 30.001475555s FAIL (2025-08-07 22:21:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001516788s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXX.X....X.X.................................XXXXXXXXXXXXXX] duration: 30.000706604s FAIL (2025-08-07 22:21:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000727057s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.........................................X............] duration: 863.471822ms FAIL (2025-08-07 22:21:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 863.492892ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.X.................................XXXXXXXXXX] duration: 1m0.001635122s FAIL (2025-08-07 22:21:45 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12h31m57.846830881s) [upload=12944]: Uploading segments... Aug 7 22:11:23.331766773 (12h31m57.846869355s) [upload=12944]: Got next segment. Inline: true Aug 7 22:11:23.331771532 (12h31m57.846874108s) [upload=12944]: Waiting for error group managing segments... Aug 7 22:11:23.331812523 (12h31m57.846915099s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:23.331817222 (12h31m57.846919788s) [upload=12944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:23.331820650 (12h31m57.846923226s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:23.301325187 (12h32m57.816427773s) [upload=12944]: Done uploading segments... Aug 7 22:12:23.301350364 (12h32m57.816452947s) [upload=12944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:33.831933438 (12h33m8.34703602s) [upload=12965]: Starting upload Aug 7 22:12:33.832018368 (12h33m8.34712093s) [upload=12965]: Done starting upload Aug 7 22:12:33.832086961 (12h33m8.347189539s) [upload=12965]: Uploading segments... Aug 7 22:12:33.832135510 (12h33m8.347238093s) [upload=12965]: Got next segment. Inline: true Aug 7 22:12:33.832141249 (12h33m8.347243826s) [upload=12965]: Waiting for error group managing segments... Aug 7 22:12:33.832194093 (12h33m8.347296674s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:33.832199847 (12h33m8.347302426s) [upload=12965]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:33.832205088 (12h33m8.347307666s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:33.802839016 (12h34m8.317941593s) [upload=12965]: Done uploading segments... Aug 7 22:13:33.802913313 (12h34m8.31801589s) [upload=12965]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.913275619 (12h34m8.428378249s) [upload=12977]: Starting upload Aug 7 22:13:33.913398357 (12h34m8.428500941s) [upload=12977]: Done starting upload Aug 7 22:13:33.913496779 (12h34m8.428599348s) [upload=12977]: Uploading segments... Aug 7 22:13:33.913568818 (12h34m8.428671424s) [upload=12977]: Got next segment. Inline: true Aug 7 22:13:33.913576485 (12h34m8.428679072s) [upload=12977]: Waiting for error group managing segments... Aug 7 22:13:33.913646668 (12h34m8.428749234s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.913652247 (12h34m8.428754834s) [upload=12977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.913655814 (12h34m8.428758404s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.874545239 (12h35m8.389647805s) [upload=12977]: Done uploading segments... Aug 7 22:14:33.874575517 (12h35m8.389678082s) [upload=12977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:38.961368058 (12h35m13.476470627s) [upload=12995]: Starting upload Aug 7 22:14:38.961455294 (12h35m13.476557856s) [upload=12995]: Done starting upload Aug 7 22:14:38.961527233 (12h35m13.476629807s) [upload=12995]: Uploading segments... Aug 7 22:14:38.961574411 (12h35m13.476676997s) [upload=12995]: Got next segment. Inline: true Aug 7 22:14:38.961580546 (12h35m13.47668312s) [upload=12995]: Waiting for error group managing segments... Aug 7 22:14:38.961644878 (12h35m13.476747457s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:38.961650553 (12h35m13.476753133s) [upload=12995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:38.961655154 (12h35m13.476757734s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:38.926876179 (12h36m13.441978762s) [upload=12995]: Done uploading segments... Aug 7 22:15:38.926898754 (12h36m13.442001334s) [upload=12995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:39.023006560 (12h36m13.538109126s) [upload=13009]: Starting upload Aug 7 22:15:39.023159911 (12h36m13.538262487s) [upload=13009]: Done starting upload Aug 7 22:15:39.023273936 (12h36m13.538376511s) [upload=13009]: Uploading segments... Aug 7 22:15:39.023358183 (12h36m13.538460767s) [upload=13009]: Got next segment. Inline: true Aug 7 22:15:39.023363552 (12h36m13.538466129s) [upload=13009]: Waiting for error group managing segments... Aug 7 22:15:39.023458469 (12h36m13.538561049s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:39.023464439 (12h36m13.538567019s) [upload=13009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:39.023491067 (12h36m13.53859365s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:38.991189889 (12h37m13.506292477s) [upload=13009]: Done uploading segments... Aug 7 22:16:38.991214272 (12h37m13.506316848s) [upload=13009]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:39.116496595 (12h37m13.631599182s) [upload=13020]: Starting upload Aug 7 22:16:39.116578790 (12h37m13.631681366s) [upload=13020]: Done starting upload Aug 7 22:16:39.116734512 (12h37m13.631837083s) [upload=13020]: Uploading segments... Aug 7 22:16:39.116797388 (12h37m13.631899986s) [upload=13020]: Got next segment. Inline: true Aug 7 22:16:39.116803839 (12h37m13.631906421s) [upload=13020]: Waiting for error group managing segments... Aug 7 22:16:39.116855539 (12h37m13.631958104s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:39.116860739 (12h37m13.631963318s) [upload=13020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:39.116864309 (12h37m13.631966875s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:39.087205419 (12h38m13.602307996s) [upload=13020]: Done uploading segments... Aug 7 22:17:39.087240451 (12h38m13.602343028s) [upload=13020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:39.195279517 (12h38m13.710382082s) [upload=13033]: Starting upload Aug 7 22:17:39.195345128 (12h38m13.710447708s) [upload=13033]: Done starting upload Aug 7 22:17:39.195408125 (12h38m13.710510701s) [upload=13033]: Uploading segments... Aug 7 22:17:39.195557815 (12h38m13.710660402s) [upload=13033]: Got next segment. Inline: true Aug 7 22:17:39.195566565 (12h38m13.710669149s) [upload=13033]: Waiting for error group managing segments... Aug 7 22:17:39.195637820 (12h38m13.710740397s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:39.195643205 (12h38m13.710745782s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:39.195646438 (12h38m13.710749014s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:39.157680356 (12h39m13.672782934s) [upload=13033]: Done uploading segments... Aug 7 22:18:39.157703138 (12h39m13.672805715s) [upload=13033]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:39.252626432 (12h39m13.767729009s) [upload=13048]: Starting upload Aug 7 22:18:39.252783205 (12h39m13.767885845s) [upload=13048]: Done starting upload Aug 7 22:18:39.252852957 (12h39m13.767955535s) [upload=13048]: Uploading segments... Aug 7 22:18:39.252949510 (12h39m13.768052096s) [upload=13048]: Got next segment. Inline: true Aug 7 22:18:39.252955069 (12h39m13.768057637s) [upload=13048]: Waiting for error group managing segments... Aug 7 22:18:39.253031088 (12h39m13.76813365s) [upload=13048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:39.253036915 (12h39m13.768139478s) [upload=13048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:39.253040017 (12h39m13.768142595s) [upload=13048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:39.223199412 (12h40m13.738302239s) [upload=13048]: Done uploading segments... Aug 7 22:19:39.223230125 (12h40m13.738332769s) [upload=13048]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:39.328604668 (12h40m13.843707247s) [upload=13065]: Starting upload Aug 7 22:19:39.328870365 (12h40m13.843972956s) [upload=13065]: Done starting upload Aug 7 22:19:39.329172418 (12h40m13.844274986s) [upload=13065]: Uploading segments... Aug 7 22:19:39.329244607 (12h40m13.844347193s) [upload=13065]: Got next segment. Inline: true Aug 7 22:19:39.329250690 (12h40m13.844353268s) [upload=13065]: Waiting for error group managing segments... Aug 7 22:19:39.329302352 (12h40m13.844404927s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:39.329309321 (12h40m13.844411902s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:39.329313964 (12h40m13.844416551s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:39.300343613 (12h41m13.81544618s) [upload=13065]: Done uploading segments... Aug 7 22:20:39.300363356 (12h41m13.815465922s) [upload=13065]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:45.259131358 (12h41m19.774233925s) [upload=13078]: Starting upload Aug 7 22:20:45.259192437 (12h41m19.774295013s) [upload=13078]: Done starting upload Aug 7 22:20:45.259251322 (12h41m19.774353897s) [upload=13078]: Uploading segments... Aug 7 22:20:45.259289829 (12h41m19.774392414s) [upload=13078]: Got next segment. Inline: true Aug 7 22:20:45.259294478 (12h41m19.774397054s) [upload=13078]: Waiting for error group managing segments... Aug 7 22:20:45.259332525 (12h41m19.774435101s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:45.259337069 (12h41m19.774439635s) [upload=13078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:45.259340263 (12h41m19.77444284s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:45.229308083 (12h42m19.744410669s) [upload=13078]: Done uploading segments... Aug 7 22:21:45.229380638 (12h42m19.744483223s) [upload=13078]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001657043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.....X..........................XXXXXXXX] duration: 1m0.000846332s FAIL (2025-08-07 22:21:36 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== :11:18.228516300 (12h31m52.743618876s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.241530379 (12h31m52.756632941s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.251935797 (12h31m52.767038375s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.277091280 (12h31m52.792193849s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291897936 (12h31m52.807000513s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291922152 (12h31m52.80702473s) [seg_pos={0 0}, upload=12943]: Segment reached optimal threshold of 14 pieces. Aug 7 22:11:18.292352474 (12h31m52.807455054s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:18.292749416 (12h31m52.807851996s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection Aug 7 22:11:18.293007428 (12h31m52.808110005s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: read tcp 159.223.143.39:34280->64.49.13.26:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293070269 (12h31m52.808172846s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: read tcp 159.223.143.39:46438->104.36.82.45:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293086568 (12h31m52.808189151s) [seg_pos={0 0}, upload=12943]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:11:18.293166574 (12h31m52.808269155s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:18.293172005 (12h31m52.808274588s) [upload=12943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:08.727892812 (12h34m43.242995378s) [upload=12943]: Done uploading segments... Aug 7 22:14:08.727919062 (12h34m43.243021627s) [upload=12943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:08.820786996 (12h34m43.335889564s) [upload=12987]: Starting upload Aug 7 22:14:08.820869466 (12h34m43.335972332s) [upload=12987]: Done starting upload Aug 7 22:14:08.820990812 (12h34m43.336093389s) [upload=12987]: Uploading segments... Aug 7 22:14:08.821180945 (12h34m43.336283523s) [upload=12987]: Got next segment. Inline: false Aug 7 22:14:08.821227735 (12h34m43.336330315s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:08.821233790 (12h34m43.336336478s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:08.788378642 (12h35m43.303481218s) [upload=12987]: Next segment err: context deadline exceeded Aug 7 22:15:08.788567212 (12h35m43.303669789s) [upload=12987]: Done uploading segments... Aug 7 22:15:08.788593806 (12h35m43.303696388s) [upload=12987]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.518494042 (12h35m57.033596616s) [upload=13004]: Starting upload Aug 7 22:15:22.518605977 (12h35m57.033708558s) [upload=13004]: Done starting upload Aug 7 22:15:22.518927572 (12h35m57.034030143s) [upload=13004]: Uploading segments... Aug 7 22:15:22.519083330 (12h35m57.034185915s) [upload=13004]: Got next segment. Inline: false Aug 7 22:15:22.519139330 (12h35m57.034241894s) [upload=13004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.519145674 (12h35m57.034248259s) [upload=13004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.486086322 (12h36m57.001188918s) [upload=13004]: Next segment err: context deadline exceeded Aug 7 22:16:22.486315849 (12h36m57.00141843s) [upload=13004]: Done uploading segments... Aug 7 22:16:22.486353471 (12h36m57.001456038s) [upload=13004]: Upload finished. err: context deadline exceeded Aug 7 22:16:26.977501497 (12h37m1.492604079s) [upload=13017]: Starting upload Aug 7 22:16:26.977634110 (12h37m1.492736693s) [upload=13017]: Done starting upload Aug 7 22:16:26.977724277 (12h37m1.492826848s) [upload=13017]: Uploading segments... Aug 7 22:16:26.978074717 (12h37m1.493177287s) [upload=13017]: Got next segment. Inline: false Aug 7 22:16:26.978159060 (12h37m1.493261638s) [upload=13017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:26.978164291 (12h37m1.493266867s) [upload=13017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:26.948529899 (12h38m1.463632483s) [upload=13017]: Next segment err: context deadline exceeded Aug 7 22:17:26.948553603 (12h38m1.463656182s) [upload=13017]: Done uploading segments... Aug 7 22:17:26.948580927 (12h38m1.463683497s) [upload=13017]: Upload finished. err: context deadline exceeded Aug 7 22:17:27.068460198 (12h38m1.583562778s) [upload=13032]: Starting upload Aug 7 22:17:27.068564049 (12h38m1.583666643s) [upload=13032]: Done starting upload Aug 7 22:17:27.068889168 (12h38m1.583991745s) [upload=13032]: Uploading segments... Aug 7 22:17:27.069219219 (12h38m1.584321786s) [upload=13032]: Got next segment. Inline: false Aug 7 22:17:27.069277883 (12h38m1.584380461s) [upload=13032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:27.069305785 (12h38m1.58440837s) [upload=13032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:27.039567178 (12h39m1.554669767s) [upload=13032]: Next segment err: context deadline exceeded Aug 7 22:18:27.039666174 (12h39m1.554768736s) [upload=13032]: Done uploading segments... Aug 7 22:18:27.039688215 (12h39m1.554790778s) [upload=13032]: Upload finished. err: context deadline exceeded Aug 7 22:18:36.777819622 (12h39m11.292922206s) [upload=13047]: Starting upload Aug 7 22:18:36.777959930 (12h39m11.293062515s) [upload=13047]: Done starting upload Aug 7 22:18:36.778221018 (12h39m11.29332361s) [upload=13047]: Uploading segments... Aug 7 22:18:36.778469733 (12h39m11.293572319s) [upload=13047]: Got next segment. Inline: false Aug 7 22:18:36.778552648 (12h39m11.293655234s) [upload=13047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:36.778559731 (12h39m11.293662318s) [upload=13047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:36.748208363 (12h40m11.263310949s) [upload=13047]: Next segment err: context deadline exceeded Aug 7 22:19:36.748420878 (12h40m11.263523459s) [upload=13047]: Done uploading segments... Aug 7 22:19:36.748450879 (12h40m11.26355346s) [upload=13047]: Upload finished. err: context deadline exceeded Aug 7 22:19:36.874412908 (12h40m11.389515476s) [upload=13064]: Starting upload Aug 7 22:19:36.874484846 (12h40m11.389587414s) [upload=13064]: Done starting upload Aug 7 22:19:36.874815576 (12h40m11.389918144s) [upload=13064]: Uploading segments... Aug 7 22:19:36.874886597 (12h40m11.389989181s) [upload=13064]: Got next segment. Inline: false Aug 7 22:19:36.874921953 (12h40m11.390024535s) [upload=13064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:36.874930610 (12h40m11.390033195s) [upload=13064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:20:36.842769930 (12h41m11.357872515s) [upload=13064]: Next segment err: context deadline exceeded Aug 7 22:20:36.842833283 (12h41m11.357935846s) [upload=13064]: Done uploading segments... Aug 7 22:20:36.842847394 (12h41m11.357949957s) [upload=13064]: Upload finished. err: context deadline exceeded Aug 7 22:20:36.950394779 (12h41m11.465497359s) [upload=13076]: Starting upload Aug 7 22:20:36.950472520 (12h41m11.4655751s) [upload=13076]: Done starting upload Aug 7 22:20:36.950562858 (12h41m11.465665423s) [upload=13076]: Uploading segments... Aug 7 22:20:36.950771505 (12h41m11.465874083s) [upload=13076]: Got next segment. Inline: false Aug 7 22:20:36.950816148 (12h41m11.465918724s) [upload=13076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:36.950820956 (12h41m11.465923532s) [upload=13076]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:36.921505604 (12h42m11.43660819s) [upload=13076]: Next segment err: context deadline exceeded Aug 7 22:21:36.921531258 (12h42m11.436633847s) [upload=13076]: Done uploading segments... Aug 7 22:21:36.921563121 (12h42m11.436665701s) [upload=13076]: Upload finished. err: context deadline exceeded Duration: 1m0.000862098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXX.XX.X..X..X..........................XXXXXXXXXXXXXX] duration: 30.000589013s FAIL (2025-08-07 22:21:30 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000611584s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........................................X...........] duration: 715.460174ms FAIL (2025-08-07 22:21:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 715.489634ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXX.XX...XXX...............................XXXXXXXXXXXXX] duration: 30.001474016s FAIL (2025-08-07 22:21:28 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:19.727995115 (12h36m54.243097692s) [upload=13016]: Starting upload Aug 7 22:16:19.728064428 (12h36m54.243167004s) [upload=13016]: Done starting upload Aug 7 22:16:19.728134488 (12h36m54.243237051s) [upload=13016]: Uploading segments... Aug 7 22:16:19.728182154 (12h36m54.243284736s) [upload=13016]: Got next segment. Inline: true Aug 7 22:16:19.728186380 (12h36m54.243288955s) [upload=13016]: Waiting for error group managing segments... Aug 7 22:16:19.728223320 (12h36m54.243325899s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:19.728227150 (12h36m54.243329727s) [upload=13016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:19.728229527 (12h36m54.243332103s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:49.691229995 (12h37m24.206332575s) [upload=13016]: Done uploading segments... Aug 7 22:16:49.691252411 (12h37m24.206354988s) [upload=13016]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:13.655336229 (12h37m48.170438797s) [upload=13028]: Starting upload Aug 7 22:17:13.655409916 (12h37m48.170512492s) [upload=13028]: Done starting upload Aug 7 22:17:13.655478262 (12h37m48.17058083s) [upload=13028]: Uploading segments... Aug 7 22:17:13.655548537 (12h37m48.17065113s) [upload=13028]: Got next segment. Inline: true Aug 7 22:17:13.655555256 (12h37m48.170657826s) [upload=13028]: Waiting for error group managing segments... Aug 7 22:17:13.655620901 (12h37m48.170723469s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:13.655627928 (12h37m48.170730508s) [upload=13028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:13.655632202 (12h37m48.170734786s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:43.624980277 (12h38m18.140082854s) [upload=13028]: Done uploading segments... Aug 7 22:17:43.624997725 (12h38m18.140100301s) [upload=13028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:43.747114121 (12h38m18.2622167s) [upload=13037]: Starting upload Aug 7 22:17:43.747186113 (12h38m18.262288692s) [upload=13037]: Done starting upload Aug 7 22:17:43.747260024 (12h38m18.262362591s) [upload=13037]: Uploading segments... Aug 7 22:17:43.747409410 (12h38m18.262511987s) [upload=13037]: Got next segment. Inline: true Aug 7 22:17:43.747414935 (12h38m18.262517512s) [upload=13037]: Waiting for error group managing segments... Aug 7 22:17:43.747457457 (12h38m18.262560036s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:43.747461997 (12h38m18.262564572s) [upload=13037]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:43.747465159 (12h38m18.262567736s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:13.715508579 (12h38m48.230611159s) [upload=13037]: Done uploading segments... Aug 7 22:18:13.715531957 (12h38m48.230634536s) [upload=13037]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:33.297944141 (12h39m7.813046721s) [upload=13046]: Starting upload Aug 7 22:18:33.298041745 (12h39m7.813144331s) [upload=13046]: Done starting upload Aug 7 22:18:33.298133963 (12h39m7.813236527s) [upload=13046]: Uploading segments... Aug 7 22:18:33.298195648 (12h39m7.813298236s) [upload=13046]: Got next segment. Inline: true Aug 7 22:18:33.298201348 (12h39m7.813303934s) [upload=13046]: Waiting for error group managing segments... Aug 7 22:18:33.298250060 (12h39m7.813352642s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:33.298255690 (12h39m7.813358273s) [upload=13046]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:33.298259809 (12h39m7.813362387s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:03.266409729 (12h39m37.781512306s) [upload=13046]: Done uploading segments... Aug 7 22:19:03.266458971 (12h39m37.781561549s) [upload=13046]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:30.020553557 (12h40m4.535656125s) [upload=13060]: Starting upload Aug 7 22:19:30.020630355 (12h40m4.535732931s) [upload=13060]: Done starting upload Aug 7 22:19:30.020870620 (12h40m4.535973198s) [upload=13060]: Uploading segments... Aug 7 22:19:30.020984540 (12h40m4.536087103s) [upload=13060]: Got next segment. Inline: true Aug 7 22:19:30.020990929 (12h40m4.536093523s) [upload=13060]: Waiting for error group managing segments... Aug 7 22:19:30.021057882 (12h40m4.536160461s) [upload=13060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:30.021063707 (12h40m4.536166287s) [upload=13060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:30.021067070 (12h40m4.536169633s) [upload=13060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:59.990140275 (12h40m34.505242847s) [upload=13060]: Done uploading segments... Aug 7 22:19:59.990166176 (12h40m34.505268763s) [upload=13060]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:14.462825235 (12h40m48.977927811s) [upload=13071]: Starting upload Aug 7 22:20:14.462946370 (12h40m48.978048965s) [upload=13071]: Done starting upload Aug 7 22:20:14.463094242 (12h40m48.978196809s) [upload=13071]: Uploading segments... Aug 7 22:20:14.463162579 (12h40m48.978265164s) [upload=13071]: Got next segment. Inline: true Aug 7 22:20:14.463168150 (12h40m48.978270729s) [upload=13071]: Waiting for error group managing segments... Aug 7 22:20:14.463216881 (12h40m48.978319457s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:14.463226562 (12h40m48.978329138s) [upload=13071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:14.463230088 (12h40m48.978332654s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:44.419309349 (12h41m18.934411917s) [upload=13071]: Done uploading segments... Aug 7 22:20:44.419327995 (12h41m18.934430565s) [upload=13071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:59.026227228 (12h41m33.541329811s) [upload=13081]: Starting upload Aug 7 22:20:59.026335834 (12h41m33.541438465s) [upload=13081]: Done starting upload Aug 7 22:20:59.026413148 (12h41m33.541515725s) [upload=13081]: Uploading segments... Aug 7 22:20:59.026487957 (12h41m33.541590544s) [upload=13081]: Got next segment. Inline: true Aug 7 22:20:59.026494660 (12h41m33.54159724s) [upload=13081]: Waiting for error group managing segments... Aug 7 22:20:59.026548283 (12h41m33.541650867s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:59.026554014 (12h41m33.541656591s) [upload=13081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:59.026557057 (12h41m33.541659621s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:28.995870666 (12h42m3.510973249s) [upload=13081]: Done uploading segments... Aug 7 22:21:28.995890051 (12h42m3.510992681s) [upload=13081]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001513294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXX.XX.X..X.............................XX] duration: 30.00123202s FAIL (2025-08-07 22:21:22 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001247543s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XX...X.............................XXXXXXXXXX] duration: 1m0.001802359s FAIL (2025-08-07 22:21:06 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001816165s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXX....XX................................XXXXXXXXXXXXXX] duration: 30.001269973s FAIL (2025-08-07 22:21:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00129282s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXX.X....X.X.................................XXXXXXXXXXXXX] duration: 30.001387522s FAIL (2025-08-07 22:20:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001399763s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXX.XX.X..X.............................X] duration: 9m55.910655047s FAIL (2025-08-07 22:20:52 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 9m55.910675367s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXX.XX...XXX...............................XXXXXXXXXXXX] duration: 30.000446215s FAIL (2025-08-07 22:20:44 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 38333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:19.727995115 (12h36m54.243097692s) [upload=13016]: Starting upload Aug 7 22:16:19.728064428 (12h36m54.243167004s) [upload=13016]: Done starting upload Aug 7 22:16:19.728134488 (12h36m54.243237051s) [upload=13016]: Uploading segments... Aug 7 22:16:19.728182154 (12h36m54.243284736s) [upload=13016]: Got next segment. Inline: true Aug 7 22:16:19.728186380 (12h36m54.243288955s) [upload=13016]: Waiting for error group managing segments... Aug 7 22:16:19.728223320 (12h36m54.243325899s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:19.728227150 (12h36m54.243329727s) [upload=13016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:19.728229527 (12h36m54.243332103s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:49.691229995 (12h37m24.206332575s) [upload=13016]: Done uploading segments... Aug 7 22:16:49.691252411 (12h37m24.206354988s) [upload=13016]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:13.655336229 (12h37m48.170438797s) [upload=13028]: Starting upload Aug 7 22:17:13.655409916 (12h37m48.170512492s) [upload=13028]: Done starting upload Aug 7 22:17:13.655478262 (12h37m48.17058083s) [upload=13028]: Uploading segments... Aug 7 22:17:13.655548537 (12h37m48.17065113s) [upload=13028]: Got next segment. Inline: true Aug 7 22:17:13.655555256 (12h37m48.170657826s) [upload=13028]: Waiting for error group managing segments... Aug 7 22:17:13.655620901 (12h37m48.170723469s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:13.655627928 (12h37m48.170730508s) [upload=13028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:13.655632202 (12h37m48.170734786s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:43.624980277 (12h38m18.140082854s) [upload=13028]: Done uploading segments... Aug 7 22:17:43.624997725 (12h38m18.140100301s) [upload=13028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:43.747114121 (12h38m18.2622167s) [upload=13037]: Starting upload Aug 7 22:17:43.747186113 (12h38m18.262288692s) [upload=13037]: Done starting upload Aug 7 22:17:43.747260024 (12h38m18.262362591s) [upload=13037]: Uploading segments... Aug 7 22:17:43.747409410 (12h38m18.262511987s) [upload=13037]: Got next segment. Inline: true Aug 7 22:17:43.747414935 (12h38m18.262517512s) [upload=13037]: Waiting for error group managing segments... Aug 7 22:17:43.747457457 (12h38m18.262560036s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:43.747461997 (12h38m18.262564572s) [upload=13037]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:43.747465159 (12h38m18.262567736s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:13.715508579 (12h38m48.230611159s) [upload=13037]: Done uploading segments... Aug 7 22:18:13.715531957 (12h38m48.230634536s) [upload=13037]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:33.297944141 (12h39m7.813046721s) [upload=13046]: Starting upload Aug 7 22:18:33.298041745 (12h39m7.813144331s) [upload=13046]: Done starting upload Aug 7 22:18:33.298133963 (12h39m7.813236527s) [upload=13046]: Uploading segments... Aug 7 22:18:33.298195648 (12h39m7.813298236s) [upload=13046]: Got next segment. Inline: true Aug 7 22:18:33.298201348 (12h39m7.813303934s) [upload=13046]: Waiting for error group managing segments... Aug 7 22:18:33.298250060 (12h39m7.813352642s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:33.298255690 (12h39m7.813358273s) [upload=13046]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:33.298259809 (12h39m7.813362387s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:03.266409729 (12h39m37.781512306s) [upload=13046]: Done uploading segments... Aug 7 22:19:03.266458971 (12h39m37.781561549s) [upload=13046]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:30.020553557 (12h40m4.535656125s) [upload=13060]: Starting upload Aug 7 22:19:30.020630355 (12h40m4.535732931s) [upload=13060]: Done starting upload Aug 7 22:19:30.020870620 (12h40m4.535973198s) [upload=13060]: Uploading segments... Aug 7 22:19:30.020984540 (12h40m4.536087103s) [upload=13060]: Got next segment. Inline: true Aug 7 22:19:30.020990929 (12h40m4.536093523s) [upload=13060]: Waiting for error group managing segments... Aug 7 22:19:30.021057882 (12h40m4.536160461s) [upload=13060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:30.021063707 (12h40m4.536166287s) [upload=13060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:30.021067070 (12h40m4.536169633s) [upload=13060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:59.990140275 (12h40m34.505242847s) [upload=13060]: Done uploading segments... Aug 7 22:19:59.990166176 (12h40m34.505268763s) [upload=13060]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:14.462825235 (12h40m48.977927811s) [upload=13071]: Starting upload Aug 7 22:20:14.462946370 (12h40m48.978048965s) [upload=13071]: Done starting upload Aug 7 22:20:14.463094242 (12h40m48.978196809s) [upload=13071]: Uploading segments... Aug 7 22:20:14.463162579 (12h40m48.978265164s) [upload=13071]: Got next segment. Inline: true Aug 7 22:20:14.463168150 (12h40m48.978270729s) [upload=13071]: Waiting for error group managing segments... Aug 7 22:20:14.463216881 (12h40m48.978319457s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:14.463226562 (12h40m48.978329138s) [upload=13071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:14.463230088 (12h40m48.978332654s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:44.419309349 (12h41m18.934411917s) [upload=13071]: Done uploading segments... Aug 7 22:20:44.419327995 (12h41m18.934430565s) [upload=13071]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000576248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.X.................................XXXXXXXXX] duration: 1m0.001247958s FAIL (2025-08-07 22:20:39 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad Aug 7 22:10:20.772231905 (12h30m55.287334472s) [upload=12928]: Uploading segments... Aug 7 22:10:20.772284367 (12h30m55.287386947s) [upload=12928]: Got next segment. Inline: true Aug 7 22:10:20.772289092 (12h30m55.287391658s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:10:20.772344988 (12h30m55.287447554s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:20.772349640 (12h30m55.287452217s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:20.772352965 (12h30m55.287455541s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:20.857003207 (12h30m55.372105787s) [upload=12928]: Done uploading segments... Aug 7 22:10:20.857032212 (12h30m55.372134783s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:11:23.331604570 (12h31m57.846707137s) [upload=12944]: Starting upload Aug 7 22:11:23.331668435 (12h31m57.846771012s) [upload=12944]: Done starting upload Aug 7 22:11:23.331728315 (12h31m57.846830881s) [upload=12944]: Uploading segments... Aug 7 22:11:23.331766773 (12h31m57.846869355s) [upload=12944]: Got next segment. Inline: true Aug 7 22:11:23.331771532 (12h31m57.846874108s) [upload=12944]: Waiting for error group managing segments... Aug 7 22:11:23.331812523 (12h31m57.846915099s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:23.331817222 (12h31m57.846919788s) [upload=12944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:23.331820650 (12h31m57.846923226s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:23.301325187 (12h32m57.816427773s) [upload=12944]: Done uploading segments... Aug 7 22:12:23.301350364 (12h32m57.816452947s) [upload=12944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:33.831933438 (12h33m8.34703602s) [upload=12965]: Starting upload Aug 7 22:12:33.832018368 (12h33m8.34712093s) [upload=12965]: Done starting upload Aug 7 22:12:33.832086961 (12h33m8.347189539s) [upload=12965]: Uploading segments... Aug 7 22:12:33.832135510 (12h33m8.347238093s) [upload=12965]: Got next segment. Inline: true Aug 7 22:12:33.832141249 (12h33m8.347243826s) [upload=12965]: Waiting for error group managing segments... Aug 7 22:12:33.832194093 (12h33m8.347296674s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:33.832199847 (12h33m8.347302426s) [upload=12965]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:33.832205088 (12h33m8.347307666s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:33.802839016 (12h34m8.317941593s) [upload=12965]: Done uploading segments... Aug 7 22:13:33.802913313 (12h34m8.31801589s) [upload=12965]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.913275619 (12h34m8.428378249s) [upload=12977]: Starting upload Aug 7 22:13:33.913398357 (12h34m8.428500941s) [upload=12977]: Done starting upload Aug 7 22:13:33.913496779 (12h34m8.428599348s) [upload=12977]: Uploading segments... Aug 7 22:13:33.913568818 (12h34m8.428671424s) [upload=12977]: Got next segment. Inline: true Aug 7 22:13:33.913576485 (12h34m8.428679072s) [upload=12977]: Waiting for error group managing segments... Aug 7 22:13:33.913646668 (12h34m8.428749234s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.913652247 (12h34m8.428754834s) [upload=12977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.913655814 (12h34m8.428758404s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.874545239 (12h35m8.389647805s) [upload=12977]: Done uploading segments... Aug 7 22:14:33.874575517 (12h35m8.389678082s) [upload=12977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:38.961368058 (12h35m13.476470627s) [upload=12995]: Starting upload Aug 7 22:14:38.961455294 (12h35m13.476557856s) [upload=12995]: Done starting upload Aug 7 22:14:38.961527233 (12h35m13.476629807s) [upload=12995]: Uploading segments... Aug 7 22:14:38.961574411 (12h35m13.476676997s) [upload=12995]: Got next segment. Inline: true Aug 7 22:14:38.961580546 (12h35m13.47668312s) [upload=12995]: Waiting for error group managing segments... Aug 7 22:14:38.961644878 (12h35m13.476747457s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:38.961650553 (12h35m13.476753133s) [upload=12995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:38.961655154 (12h35m13.476757734s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:38.926876179 (12h36m13.441978762s) [upload=12995]: Done uploading segments... Aug 7 22:15:38.926898754 (12h36m13.442001334s) [upload=12995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:39.023006560 (12h36m13.538109126s) [upload=13009]: Starting upload Aug 7 22:15:39.023159911 (12h36m13.538262487s) [upload=13009]: Done starting upload Aug 7 22:15:39.023273936 (12h36m13.538376511s) [upload=13009]: Uploading segments... Aug 7 22:15:39.023358183 (12h36m13.538460767s) [upload=13009]: Got next segment. Inline: true Aug 7 22:15:39.023363552 (12h36m13.538466129s) [upload=13009]: Waiting for error group managing segments... Aug 7 22:15:39.023458469 (12h36m13.538561049s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:39.023464439 (12h36m13.538567019s) [upload=13009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:39.023491067 (12h36m13.53859365s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:38.991189889 (12h37m13.506292477s) [upload=13009]: Done uploading segments... Aug 7 22:16:38.991214272 (12h37m13.506316848s) [upload=13009]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:39.116496595 (12h37m13.631599182s) [upload=13020]: Starting upload Aug 7 22:16:39.116578790 (12h37m13.631681366s) [upload=13020]: Done starting upload Aug 7 22:16:39.116734512 (12h37m13.631837083s) [upload=13020]: Uploading segments... Aug 7 22:16:39.116797388 (12h37m13.631899986s) [upload=13020]: Got next segment. Inline: true Aug 7 22:16:39.116803839 (12h37m13.631906421s) [upload=13020]: Waiting for error group managing segments... Aug 7 22:16:39.116855539 (12h37m13.631958104s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:39.116860739 (12h37m13.631963318s) [upload=13020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:39.116864309 (12h37m13.631966875s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:39.087205419 (12h38m13.602307996s) [upload=13020]: Done uploading segments... Aug 7 22:17:39.087240451 (12h38m13.602343028s) [upload=13020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:39.195279517 (12h38m13.710382082s) [upload=13033]: Starting upload Aug 7 22:17:39.195345128 (12h38m13.710447708s) [upload=13033]: Done starting upload Aug 7 22:17:39.195408125 (12h38m13.710510701s) [upload=13033]: Uploading segments... Aug 7 22:17:39.195557815 (12h38m13.710660402s) [upload=13033]: Got next segment. Inline: true Aug 7 22:17:39.195566565 (12h38m13.710669149s) [upload=13033]: Waiting for error group managing segments... Aug 7 22:17:39.195637820 (12h38m13.710740397s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:39.195643205 (12h38m13.710745782s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:39.195646438 (12h38m13.710749014s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:39.157680356 (12h39m13.672782934s) [upload=13033]: Done uploading segments... Aug 7 22:18:39.157703138 (12h39m13.672805715s) [upload=13033]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:39.252626432 (12h39m13.767729009s) [upload=13048]: Starting upload Aug 7 22:18:39.252783205 (12h39m13.767885845s) [upload=13048]: Done starting upload Aug 7 22:18:39.252852957 (12h39m13.767955535s) [upload=13048]: Uploading segments... Aug 7 22:18:39.252949510 (12h39m13.768052096s) [upload=13048]: Got next segment. Inline: true Aug 7 22:18:39.252955069 (12h39m13.768057637s) [upload=13048]: Waiting for error group managing segments... Aug 7 22:18:39.253031088 (12h39m13.76813365s) [upload=13048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:39.253036915 (12h39m13.768139478s) [upload=13048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:39.253040017 (12h39m13.768142595s) [upload=13048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:39.223199412 (12h40m13.738302239s) [upload=13048]: Done uploading segments... Aug 7 22:19:39.223230125 (12h40m13.738332769s) [upload=13048]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:39.328604668 (12h40m13.843707247s) [upload=13065]: Starting upload Aug 7 22:19:39.328870365 (12h40m13.843972956s) [upload=13065]: Done starting upload Aug 7 22:19:39.329172418 (12h40m13.844274986s) [upload=13065]: Uploading segments... Aug 7 22:19:39.329244607 (12h40m13.844347193s) [upload=13065]: Got next segment. Inline: true Aug 7 22:19:39.329250690 (12h40m13.844353268s) [upload=13065]: Waiting for error group managing segments... Aug 7 22:19:39.329302352 (12h40m13.844404927s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:39.329309321 (12h40m13.844411902s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:39.329313964 (12h40m13.844416551s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:39.300343613 (12h41m13.81544618s) [upload=13065]: Done uploading segments... Aug 7 22:20:39.300363356 (12h41m13.815465922s) [upload=13065]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001372109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXX.XX.X..X..X..........................XXXXXXXXXXXXX] duration: 30.000555483s FAIL (2025-08-07 22:20:38 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000576457s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXX.....X..........................XXXXXXX] duration: 1m0.001108473s FAIL (2025-08-07 22:20:36 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 7 22:11:18.217895630 (12h31m52.732998217s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.218622391 (12h31m52.733724974s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.219619423 (12h31m52.734722004s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.227715032 (12h31m52.742817609s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.228516300 (12h31m52.743618876s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.241530379 (12h31m52.756632941s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.251935797 (12h31m52.767038375s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.277091280 (12h31m52.792193849s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291897936 (12h31m52.807000513s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291922152 (12h31m52.80702473s) [seg_pos={0 0}, upload=12943]: Segment reached optimal threshold of 14 pieces. Aug 7 22:11:18.292352474 (12h31m52.807455054s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:18.292749416 (12h31m52.807851996s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection Aug 7 22:11:18.293007428 (12h31m52.808110005s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: read tcp 159.223.143.39:34280->64.49.13.26:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293070269 (12h31m52.808172846s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: read tcp 159.223.143.39:46438->104.36.82.45:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293086568 (12h31m52.808189151s) [seg_pos={0 0}, upload=12943]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:11:18.293166574 (12h31m52.808269155s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:18.293172005 (12h31m52.808274588s) [upload=12943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:08.727892812 (12h34m43.242995378s) [upload=12943]: Done uploading segments... Aug 7 22:14:08.727919062 (12h34m43.243021627s) [upload=12943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:08.820786996 (12h34m43.335889564s) [upload=12987]: Starting upload Aug 7 22:14:08.820869466 (12h34m43.335972332s) [upload=12987]: Done starting upload Aug 7 22:14:08.820990812 (12h34m43.336093389s) [upload=12987]: Uploading segments... Aug 7 22:14:08.821180945 (12h34m43.336283523s) [upload=12987]: Got next segment. Inline: false Aug 7 22:14:08.821227735 (12h34m43.336330315s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:08.821233790 (12h34m43.336336478s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:08.788378642 (12h35m43.303481218s) [upload=12987]: Next segment err: context deadline exceeded Aug 7 22:15:08.788567212 (12h35m43.303669789s) [upload=12987]: Done uploading segments... Aug 7 22:15:08.788593806 (12h35m43.303696388s) [upload=12987]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.518494042 (12h35m57.033596616s) [upload=13004]: Starting upload Aug 7 22:15:22.518605977 (12h35m57.033708558s) [upload=13004]: Done starting upload Aug 7 22:15:22.518927572 (12h35m57.034030143s) [upload=13004]: Uploading segments... Aug 7 22:15:22.519083330 (12h35m57.034185915s) [upload=13004]: Got next segment. Inline: false Aug 7 22:15:22.519139330 (12h35m57.034241894s) [upload=13004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.519145674 (12h35m57.034248259s) [upload=13004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.486086322 (12h36m57.001188918s) [upload=13004]: Next segment err: context deadline exceeded Aug 7 22:16:22.486315849 (12h36m57.00141843s) [upload=13004]: Done uploading segments... Aug 7 22:16:22.486353471 (12h36m57.001456038s) [upload=13004]: Upload finished. err: context deadline exceeded Aug 7 22:16:26.977501497 (12h37m1.492604079s) [upload=13017]: Starting upload Aug 7 22:16:26.977634110 (12h37m1.492736693s) [upload=13017]: Done starting upload Aug 7 22:16:26.977724277 (12h37m1.492826848s) [upload=13017]: Uploading segments... Aug 7 22:16:26.978074717 (12h37m1.493177287s) [upload=13017]: Got next segment. Inline: false Aug 7 22:16:26.978159060 (12h37m1.493261638s) [upload=13017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:26.978164291 (12h37m1.493266867s) [upload=13017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:26.948529899 (12h38m1.463632483s) [upload=13017]: Next segment err: context deadline exceeded Aug 7 22:17:26.948553603 (12h38m1.463656182s) [upload=13017]: Done uploading segments... Aug 7 22:17:26.948580927 (12h38m1.463683497s) [upload=13017]: Upload finished. err: context deadline exceeded Aug 7 22:17:27.068460198 (12h38m1.583562778s) [upload=13032]: Starting upload Aug 7 22:17:27.068564049 (12h38m1.583666643s) [upload=13032]: Done starting upload Aug 7 22:17:27.068889168 (12h38m1.583991745s) [upload=13032]: Uploading segments... Aug 7 22:17:27.069219219 (12h38m1.584321786s) [upload=13032]: Got next segment. Inline: false Aug 7 22:17:27.069277883 (12h38m1.584380461s) [upload=13032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:27.069305785 (12h38m1.58440837s) [upload=13032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:27.039567178 (12h39m1.554669767s) [upload=13032]: Next segment err: context deadline exceeded Aug 7 22:18:27.039666174 (12h39m1.554768736s) [upload=13032]: Done uploading segments... Aug 7 22:18:27.039688215 (12h39m1.554790778s) [upload=13032]: Upload finished. err: context deadline exceeded Aug 7 22:18:36.777819622 (12h39m11.292922206s) [upload=13047]: Starting upload Aug 7 22:18:36.777959930 (12h39m11.293062515s) [upload=13047]: Done starting upload Aug 7 22:18:36.778221018 (12h39m11.29332361s) [upload=13047]: Uploading segments... Aug 7 22:18:36.778469733 (12h39m11.293572319s) [upload=13047]: Got next segment. Inline: false Aug 7 22:18:36.778552648 (12h39m11.293655234s) [upload=13047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:36.778559731 (12h39m11.293662318s) [upload=13047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:36.748208363 (12h40m11.263310949s) [upload=13047]: Next segment err: context deadline exceeded Aug 7 22:19:36.748420878 (12h40m11.263523459s) [upload=13047]: Done uploading segments... Aug 7 22:19:36.748450879 (12h40m11.26355346s) [upload=13047]: Upload finished. err: context deadline exceeded Aug 7 22:19:36.874412908 (12h40m11.389515476s) [upload=13064]: Starting upload Aug 7 22:19:36.874484846 (12h40m11.389587414s) [upload=13064]: Done starting upload Aug 7 22:19:36.874815576 (12h40m11.389918144s) [upload=13064]: Uploading segments... Aug 7 22:19:36.874886597 (12h40m11.389989181s) [upload=13064]: Got next segment. Inline: false Aug 7 22:19:36.874921953 (12h40m11.390024535s) [upload=13064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:36.874930610 (12h40m11.390033195s) [upload=13064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:20:36.842769930 (12h41m11.357872515s) [upload=13064]: Next segment err: context deadline exceeded Aug 7 22:20:36.842833283 (12h41m11.357935846s) [upload=13064]: Done uploading segments... Aug 7 22:20:36.842847394 (12h41m11.357949957s) [upload=13064]: Upload finished. err: context deadline exceeded Duration: 1m0.001126289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........................................X..........] duration: 1.312857428s FAIL (2025-08-07 22:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 1.312906664s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 6 out of 60 is failed (threshold=3) [..XX..XX................................................X..X] duration: 2m0.000538186s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:20:29 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.000561852s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXX.X....X.X.................................XXXXXXXXXXXX] duration: 30.001645567s FAIL (2025-08-07 22:20:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001695234s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXX....XX................................XXXXXXXXXXXXX] duration: 30.001446483s FAIL (2025-08-07 22:20:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001471433s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XX...X.............................XXXXXXXXX] duration: 1m0.001038747s FAIL (2025-08-07 22:20:06 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001052466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XX.X..X..X..........................XXXXXXXXXXXX] duration: 30.000735429s FAIL (2025-08-07 22:20:00 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000758749s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXX.XX...XXX...............................XXXXXXXXXXX] duration: 30.000668233s FAIL (2025-08-07 22:19:59 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:19.727995115 (12h36m54.243097692s) [upload=13016]: Starting upload Aug 7 22:16:19.728064428 (12h36m54.243167004s) [upload=13016]: Done starting upload Aug 7 22:16:19.728134488 (12h36m54.243237051s) [upload=13016]: Uploading segments... Aug 7 22:16:19.728182154 (12h36m54.243284736s) [upload=13016]: Got next segment. Inline: true Aug 7 22:16:19.728186380 (12h36m54.243288955s) [upload=13016]: Waiting for error group managing segments... Aug 7 22:16:19.728223320 (12h36m54.243325899s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:19.728227150 (12h36m54.243329727s) [upload=13016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:19.728229527 (12h36m54.243332103s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:49.691229995 (12h37m24.206332575s) [upload=13016]: Done uploading segments... Aug 7 22:16:49.691252411 (12h37m24.206354988s) [upload=13016]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:13.655336229 (12h37m48.170438797s) [upload=13028]: Starting upload Aug 7 22:17:13.655409916 (12h37m48.170512492s) [upload=13028]: Done starting upload Aug 7 22:17:13.655478262 (12h37m48.17058083s) [upload=13028]: Uploading segments... Aug 7 22:17:13.655548537 (12h37m48.17065113s) [upload=13028]: Got next segment. Inline: true Aug 7 22:17:13.655555256 (12h37m48.170657826s) [upload=13028]: Waiting for error group managing segments... Aug 7 22:17:13.655620901 (12h37m48.170723469s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:13.655627928 (12h37m48.170730508s) [upload=13028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:13.655632202 (12h37m48.170734786s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:43.624980277 (12h38m18.140082854s) [upload=13028]: Done uploading segments... Aug 7 22:17:43.624997725 (12h38m18.140100301s) [upload=13028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:43.747114121 (12h38m18.2622167s) [upload=13037]: Starting upload Aug 7 22:17:43.747186113 (12h38m18.262288692s) [upload=13037]: Done starting upload Aug 7 22:17:43.747260024 (12h38m18.262362591s) [upload=13037]: Uploading segments... Aug 7 22:17:43.747409410 (12h38m18.262511987s) [upload=13037]: Got next segment. Inline: true Aug 7 22:17:43.747414935 (12h38m18.262517512s) [upload=13037]: Waiting for error group managing segments... Aug 7 22:17:43.747457457 (12h38m18.262560036s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:43.747461997 (12h38m18.262564572s) [upload=13037]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:43.747465159 (12h38m18.262567736s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:13.715508579 (12h38m48.230611159s) [upload=13037]: Done uploading segments... Aug 7 22:18:13.715531957 (12h38m48.230634536s) [upload=13037]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:33.297944141 (12h39m7.813046721s) [upload=13046]: Starting upload Aug 7 22:18:33.298041745 (12h39m7.813144331s) [upload=13046]: Done starting upload Aug 7 22:18:33.298133963 (12h39m7.813236527s) [upload=13046]: Uploading segments... Aug 7 22:18:33.298195648 (12h39m7.813298236s) [upload=13046]: Got next segment. Inline: true Aug 7 22:18:33.298201348 (12h39m7.813303934s) [upload=13046]: Waiting for error group managing segments... Aug 7 22:18:33.298250060 (12h39m7.813352642s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:33.298255690 (12h39m7.813358273s) [upload=13046]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:33.298259809 (12h39m7.813362387s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:03.266409729 (12h39m37.781512306s) [upload=13046]: Done uploading segments... Aug 7 22:19:03.266458971 (12h39m37.781561549s) [upload=13046]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:30.020553557 (12h40m4.535656125s) [upload=13060]: Starting upload Aug 7 22:19:30.020630355 (12h40m4.535732931s) [upload=13060]: Done starting upload Aug 7 22:19:30.020870620 (12h40m4.535973198s) [upload=13060]: Uploading segments... Aug 7 22:19:30.020984540 (12h40m4.536087103s) [upload=13060]: Got next segment. Inline: true Aug 7 22:19:30.020990929 (12h40m4.536093523s) [upload=13060]: Waiting for error group managing segments... Aug 7 22:19:30.021057882 (12h40m4.536160461s) [upload=13060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:30.021063707 (12h40m4.536166287s) [upload=13060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:30.021067070 (12h40m4.536169633s) [upload=13060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:59.990140275 (12h40m34.505242847s) [upload=13060]: Done uploading segments... Aug 7 22:19:59.990166176 (12h40m34.505268763s) [upload=13060]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000698786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........................................X.........] duration: 1.021614674s FAIL (2025-08-07 22:19:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 1.021638593s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXX....XX................................XXXXXXXXXXXX] duration: 30.000618741s FAIL (2025-08-07 22:19:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000637962s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.X.................................XXXXXXXX] duration: 1m0.002415799s FAIL (2025-08-07 22:19:39 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2911s) [upload=12913]: Done starting upload Aug 7 22:09:33.459666234 (12h30m7.974768811s) [upload=12913]: Uploading segments... Aug 7 22:09:33.459731513 (12h30m7.974834099s) [upload=12913]: Got next segment. Inline: true Aug 7 22:09:33.459737908 (12h30m7.974840491s) [upload=12913]: Waiting for error group managing segments... Aug 7 22:09:33.459810629 (12h30m7.974913195s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:33.459816521 (12h30m7.974919097s) [upload=12913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:33.459819752 (12h30m7.974922328s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:33.522052292 (12h30m8.037154862s) [upload=12913]: Done uploading segments... Aug 7 22:09:33.522072074 (12h30m8.037174651s) [upload=12913]: Upload finished. err: <nil> Aug 7 22:10:20.772089130 (12h30m55.287191696s) [upload=12928]: Starting upload Aug 7 22:10:20.772161634 (12h30m55.28726421s) [upload=12928]: Done starting upload Aug 7 22:10:20.772231905 (12h30m55.287334472s) [upload=12928]: Uploading segments... Aug 7 22:10:20.772284367 (12h30m55.287386947s) [upload=12928]: Got next segment. Inline: true Aug 7 22:10:20.772289092 (12h30m55.287391658s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:10:20.772344988 (12h30m55.287447554s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:20.772349640 (12h30m55.287452217s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:20.772352965 (12h30m55.287455541s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:20.857003207 (12h30m55.372105787s) [upload=12928]: Done uploading segments... Aug 7 22:10:20.857032212 (12h30m55.372134783s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:11:23.331604570 (12h31m57.846707137s) [upload=12944]: Starting upload Aug 7 22:11:23.331668435 (12h31m57.846771012s) [upload=12944]: Done starting upload Aug 7 22:11:23.331728315 (12h31m57.846830881s) [upload=12944]: Uploading segments... Aug 7 22:11:23.331766773 (12h31m57.846869355s) [upload=12944]: Got next segment. Inline: true Aug 7 22:11:23.331771532 (12h31m57.846874108s) [upload=12944]: Waiting for error group managing segments... Aug 7 22:11:23.331812523 (12h31m57.846915099s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:23.331817222 (12h31m57.846919788s) [upload=12944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:23.331820650 (12h31m57.846923226s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:23.301325187 (12h32m57.816427773s) [upload=12944]: Done uploading segments... Aug 7 22:12:23.301350364 (12h32m57.816452947s) [upload=12944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:33.831933438 (12h33m8.34703602s) [upload=12965]: Starting upload Aug 7 22:12:33.832018368 (12h33m8.34712093s) [upload=12965]: Done starting upload Aug 7 22:12:33.832086961 (12h33m8.347189539s) [upload=12965]: Uploading segments... Aug 7 22:12:33.832135510 (12h33m8.347238093s) [upload=12965]: Got next segment. Inline: true Aug 7 22:12:33.832141249 (12h33m8.347243826s) [upload=12965]: Waiting for error group managing segments... Aug 7 22:12:33.832194093 (12h33m8.347296674s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:33.832199847 (12h33m8.347302426s) [upload=12965]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:33.832205088 (12h33m8.347307666s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:33.802839016 (12h34m8.317941593s) [upload=12965]: Done uploading segments... Aug 7 22:13:33.802913313 (12h34m8.31801589s) [upload=12965]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.913275619 (12h34m8.428378249s) [upload=12977]: Starting upload Aug 7 22:13:33.913398357 (12h34m8.428500941s) [upload=12977]: Done starting upload Aug 7 22:13:33.913496779 (12h34m8.428599348s) [upload=12977]: Uploading segments... Aug 7 22:13:33.913568818 (12h34m8.428671424s) [upload=12977]: Got next segment. Inline: true Aug 7 22:13:33.913576485 (12h34m8.428679072s) [upload=12977]: Waiting for error group managing segments... Aug 7 22:13:33.913646668 (12h34m8.428749234s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.913652247 (12h34m8.428754834s) [upload=12977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.913655814 (12h34m8.428758404s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.874545239 (12h35m8.389647805s) [upload=12977]: Done uploading segments... Aug 7 22:14:33.874575517 (12h35m8.389678082s) [upload=12977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:38.961368058 (12h35m13.476470627s) [upload=12995]: Starting upload Aug 7 22:14:38.961455294 (12h35m13.476557856s) [upload=12995]: Done starting upload Aug 7 22:14:38.961527233 (12h35m13.476629807s) [upload=12995]: Uploading segments... Aug 7 22:14:38.961574411 (12h35m13.476676997s) [upload=12995]: Got next segment. Inline: true Aug 7 22:14:38.961580546 (12h35m13.47668312s) [upload=12995]: Waiting for error group managing segments... Aug 7 22:14:38.961644878 (12h35m13.476747457s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:38.961650553 (12h35m13.476753133s) [upload=12995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:38.961655154 (12h35m13.476757734s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:38.926876179 (12h36m13.441978762s) [upload=12995]: Done uploading segments... Aug 7 22:15:38.926898754 (12h36m13.442001334s) [upload=12995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:39.023006560 (12h36m13.538109126s) [upload=13009]: Starting upload Aug 7 22:15:39.023159911 (12h36m13.538262487s) [upload=13009]: Done starting upload Aug 7 22:15:39.023273936 (12h36m13.538376511s) [upload=13009]: Uploading segments... Aug 7 22:15:39.023358183 (12h36m13.538460767s) [upload=13009]: Got next segment. Inline: true Aug 7 22:15:39.023363552 (12h36m13.538466129s) [upload=13009]: Waiting for error group managing segments... Aug 7 22:15:39.023458469 (12h36m13.538561049s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:39.023464439 (12h36m13.538567019s) [upload=13009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:39.023491067 (12h36m13.53859365s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:38.991189889 (12h37m13.506292477s) [upload=13009]: Done uploading segments... Aug 7 22:16:38.991214272 (12h37m13.506316848s) [upload=13009]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:39.116496595 (12h37m13.631599182s) [upload=13020]: Starting upload Aug 7 22:16:39.116578790 (12h37m13.631681366s) [upload=13020]: Done starting upload Aug 7 22:16:39.116734512 (12h37m13.631837083s) [upload=13020]: Uploading segments... Aug 7 22:16:39.116797388 (12h37m13.631899986s) [upload=13020]: Got next segment. Inline: true Aug 7 22:16:39.116803839 (12h37m13.631906421s) [upload=13020]: Waiting for error group managing segments... Aug 7 22:16:39.116855539 (12h37m13.631958104s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:39.116860739 (12h37m13.631963318s) [upload=13020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:39.116864309 (12h37m13.631966875s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:39.087205419 (12h38m13.602307996s) [upload=13020]: Done uploading segments... Aug 7 22:17:39.087240451 (12h38m13.602343028s) [upload=13020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:39.195279517 (12h38m13.710382082s) [upload=13033]: Starting upload Aug 7 22:17:39.195345128 (12h38m13.710447708s) [upload=13033]: Done starting upload Aug 7 22:17:39.195408125 (12h38m13.710510701s) [upload=13033]: Uploading segments... Aug 7 22:17:39.195557815 (12h38m13.710660402s) [upload=13033]: Got next segment. Inline: true Aug 7 22:17:39.195566565 (12h38m13.710669149s) [upload=13033]: Waiting for error group managing segments... Aug 7 22:17:39.195637820 (12h38m13.710740397s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:39.195643205 (12h38m13.710745782s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:39.195646438 (12h38m13.710749014s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:39.157680356 (12h39m13.672782934s) [upload=13033]: Done uploading segments... Aug 7 22:18:39.157703138 (12h39m13.672805715s) [upload=13033]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:39.252626432 (12h39m13.767729009s) [upload=13048]: Starting upload Aug 7 22:18:39.252783205 (12h39m13.767885845s) [upload=13048]: Done starting upload Aug 7 22:18:39.252852957 (12h39m13.767955535s) [upload=13048]: Uploading segments... Aug 7 22:18:39.252949510 (12h39m13.768052096s) [upload=13048]: Got next segment. Inline: true Aug 7 22:18:39.252955069 (12h39m13.768057637s) [upload=13048]: Waiting for error group managing segments... Aug 7 22:18:39.253031088 (12h39m13.76813365s) [upload=13048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:39.253036915 (12h39m13.768139478s) [upload=13048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:39.253040017 (12h39m13.768142595s) [upload=13048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:39.223199412 (12h40m13.738302239s) [upload=13048]: Done uploading segments... Aug 7 22:19:39.223230125 (12h40m13.738332769s) [upload=13048]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.002450569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXX.....X..........................XXXXXX] duration: 1m0.0008523s FAIL (2025-08-07 22:19:36 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== e. err: <nil> Aug 7 22:11:18.201535099 (12h31m52.716637672s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.212513462 (12h31m52.727616038s) [noise=true , address=104.36.82.47:4552 , piece_id=NJTH3OJMQQBV6PEADF4IFYZ3U7QZHJIU774CEPVY5BBBXI6V53QA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.213955699 (12h31m52.729058282s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.214449276 (12h31m52.729551846s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.217895630 (12h31m52.732998217s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.218622391 (12h31m52.733724974s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.219619423 (12h31m52.734722004s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.227715032 (12h31m52.742817609s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.228516300 (12h31m52.743618876s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.241530379 (12h31m52.756632941s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.251935797 (12h31m52.767038375s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.277091280 (12h31m52.792193849s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291897936 (12h31m52.807000513s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291922152 (12h31m52.80702473s) [seg_pos={0 0}, upload=12943]: Segment reached optimal threshold of 14 pieces. Aug 7 22:11:18.292352474 (12h31m52.807455054s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:18.292749416 (12h31m52.807851996s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection Aug 7 22:11:18.293007428 (12h31m52.808110005s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: read tcp 159.223.143.39:34280->64.49.13.26:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293070269 (12h31m52.808172846s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: read tcp 159.223.143.39:46438->104.36.82.45:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293086568 (12h31m52.808189151s) [seg_pos={0 0}, upload=12943]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:11:18.293166574 (12h31m52.808269155s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:18.293172005 (12h31m52.808274588s) [upload=12943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:08.727892812 (12h34m43.242995378s) [upload=12943]: Done uploading segments... Aug 7 22:14:08.727919062 (12h34m43.243021627s) [upload=12943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:08.820786996 (12h34m43.335889564s) [upload=12987]: Starting upload Aug 7 22:14:08.820869466 (12h34m43.335972332s) [upload=12987]: Done starting upload Aug 7 22:14:08.820990812 (12h34m43.336093389s) [upload=12987]: Uploading segments... Aug 7 22:14:08.821180945 (12h34m43.336283523s) [upload=12987]: Got next segment. Inline: false Aug 7 22:14:08.821227735 (12h34m43.336330315s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:08.821233790 (12h34m43.336336478s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:08.788378642 (12h35m43.303481218s) [upload=12987]: Next segment err: context deadline exceeded Aug 7 22:15:08.788567212 (12h35m43.303669789s) [upload=12987]: Done uploading segments... Aug 7 22:15:08.788593806 (12h35m43.303696388s) [upload=12987]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.518494042 (12h35m57.033596616s) [upload=13004]: Starting upload Aug 7 22:15:22.518605977 (12h35m57.033708558s) [upload=13004]: Done starting upload Aug 7 22:15:22.518927572 (12h35m57.034030143s) [upload=13004]: Uploading segments... Aug 7 22:15:22.519083330 (12h35m57.034185915s) [upload=13004]: Got next segment. Inline: false Aug 7 22:15:22.519139330 (12h35m57.034241894s) [upload=13004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.519145674 (12h35m57.034248259s) [upload=13004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.486086322 (12h36m57.001188918s) [upload=13004]: Next segment err: context deadline exceeded Aug 7 22:16:22.486315849 (12h36m57.00141843s) [upload=13004]: Done uploading segments... Aug 7 22:16:22.486353471 (12h36m57.001456038s) [upload=13004]: Upload finished. err: context deadline exceeded Aug 7 22:16:26.977501497 (12h37m1.492604079s) [upload=13017]: Starting upload Aug 7 22:16:26.977634110 (12h37m1.492736693s) [upload=13017]: Done starting upload Aug 7 22:16:26.977724277 (12h37m1.492826848s) [upload=13017]: Uploading segments... Aug 7 22:16:26.978074717 (12h37m1.493177287s) [upload=13017]: Got next segment. Inline: false Aug 7 22:16:26.978159060 (12h37m1.493261638s) [upload=13017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:26.978164291 (12h37m1.493266867s) [upload=13017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:26.948529899 (12h38m1.463632483s) [upload=13017]: Next segment err: context deadline exceeded Aug 7 22:17:26.948553603 (12h38m1.463656182s) [upload=13017]: Done uploading segments... Aug 7 22:17:26.948580927 (12h38m1.463683497s) [upload=13017]: Upload finished. err: context deadline exceeded Aug 7 22:17:27.068460198 (12h38m1.583562778s) [upload=13032]: Starting upload Aug 7 22:17:27.068564049 (12h38m1.583666643s) [upload=13032]: Done starting upload Aug 7 22:17:27.068889168 (12h38m1.583991745s) [upload=13032]: Uploading segments... Aug 7 22:17:27.069219219 (12h38m1.584321786s) [upload=13032]: Got next segment. Inline: false Aug 7 22:17:27.069277883 (12h38m1.584380461s) [upload=13032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:27.069305785 (12h38m1.58440837s) [upload=13032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:27.039567178 (12h39m1.554669767s) [upload=13032]: Next segment err: context deadline exceeded Aug 7 22:18:27.039666174 (12h39m1.554768736s) [upload=13032]: Done uploading segments... Aug 7 22:18:27.039688215 (12h39m1.554790778s) [upload=13032]: Upload finished. err: context deadline exceeded Aug 7 22:18:36.777819622 (12h39m11.292922206s) [upload=13047]: Starting upload Aug 7 22:18:36.777959930 (12h39m11.293062515s) [upload=13047]: Done starting upload Aug 7 22:18:36.778221018 (12h39m11.29332361s) [upload=13047]: Uploading segments... Aug 7 22:18:36.778469733 (12h39m11.293572319s) [upload=13047]: Got next segment. Inline: false Aug 7 22:18:36.778552648 (12h39m11.293655234s) [upload=13047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:36.778559731 (12h39m11.293662318s) [upload=13047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:19:36.748208363 (12h40m11.263310949s) [upload=13047]: Next segment err: context deadline exceeded Aug 7 22:19:36.748420878 (12h40m11.263523459s) [upload=13047]: Done uploading segments... Aug 7 22:19:36.748450879 (12h40m11.26355346s) [upload=13047]: Upload finished. err: context deadline exceeded Duration: 1m0.000878162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XX.X..X..X..........................XXXXXXXXXXX] duration: 30.000765127s FAIL (2025-08-07 22:19:24 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000790377s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXX.X....X.X.................................XXXXXXXXXXX] duration: 30.00028191s FAIL (2025-08-07 22:19:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000314254s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........................................X........] duration: 976.069475ms FAIL (2025-08-07 22:19:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 976.099258ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXX....XX................................XXXXXXXXXXX] duration: 30.001419115s FAIL (2025-08-07 22:19:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001440717s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XX...X.............................XXXXXXXX] duration: 1m0.001113032s FAIL (2025-08-07 22:19:06 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001128583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXX.XX...XXX...............................XXXXXXXXXX] duration: 30.000592638s FAIL (2025-08-07 22:19:03 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37521996 (12h32m2.852624566s) [upload=12946]: Uploading segments... Aug 7 22:11:28.337560362 (12h32m2.852662948s) [upload=12946]: Got next segment. Inline: true Aug 7 22:11:28.337565059 (12h32m2.852667634s) [upload=12946]: Waiting for error group managing segments... Aug 7 22:11:28.337604389 (12h32m2.852706965s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:28.337608838 (12h32m2.852711414s) [upload=12946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:28.337612338 (12h32m2.852714915s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:58.307179904 (12h32m32.82228248s) [upload=12946]: Done uploading segments... Aug 7 22:11:58.307198958 (12h32m32.822301534s) [upload=12946]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:21.223007637 (12h32m55.738110218s) [upload=12959]: Starting upload Aug 7 22:12:21.223121202 (12h32m55.738223785s) [upload=12959]: Done starting upload Aug 7 22:12:21.223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:19.727995115 (12h36m54.243097692s) [upload=13016]: Starting upload Aug 7 22:16:19.728064428 (12h36m54.243167004s) [upload=13016]: Done starting upload Aug 7 22:16:19.728134488 (12h36m54.243237051s) [upload=13016]: Uploading segments... Aug 7 22:16:19.728182154 (12h36m54.243284736s) [upload=13016]: Got next segment. Inline: true Aug 7 22:16:19.728186380 (12h36m54.243288955s) [upload=13016]: Waiting for error group managing segments... Aug 7 22:16:19.728223320 (12h36m54.243325899s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:19.728227150 (12h36m54.243329727s) [upload=13016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:19.728229527 (12h36m54.243332103s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:49.691229995 (12h37m24.206332575s) [upload=13016]: Done uploading segments... Aug 7 22:16:49.691252411 (12h37m24.206354988s) [upload=13016]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:13.655336229 (12h37m48.170438797s) [upload=13028]: Starting upload Aug 7 22:17:13.655409916 (12h37m48.170512492s) [upload=13028]: Done starting upload Aug 7 22:17:13.655478262 (12h37m48.17058083s) [upload=13028]: Uploading segments... Aug 7 22:17:13.655548537 (12h37m48.17065113s) [upload=13028]: Got next segment. Inline: true Aug 7 22:17:13.655555256 (12h37m48.170657826s) [upload=13028]: Waiting for error group managing segments... Aug 7 22:17:13.655620901 (12h37m48.170723469s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:13.655627928 (12h37m48.170730508s) [upload=13028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:13.655632202 (12h37m48.170734786s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:43.624980277 (12h38m18.140082854s) [upload=13028]: Done uploading segments... Aug 7 22:17:43.624997725 (12h38m18.140100301s) [upload=13028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:43.747114121 (12h38m18.2622167s) [upload=13037]: Starting upload Aug 7 22:17:43.747186113 (12h38m18.262288692s) [upload=13037]: Done starting upload Aug 7 22:17:43.747260024 (12h38m18.262362591s) [upload=13037]: Uploading segments... Aug 7 22:17:43.747409410 (12h38m18.262511987s) [upload=13037]: Got next segment. Inline: true Aug 7 22:17:43.747414935 (12h38m18.262517512s) [upload=13037]: Waiting for error group managing segments... Aug 7 22:17:43.747457457 (12h38m18.262560036s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:43.747461997 (12h38m18.262564572s) [upload=13037]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:43.747465159 (12h38m18.262567736s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:13.715508579 (12h38m48.230611159s) [upload=13037]: Done uploading segments... Aug 7 22:18:13.715531957 (12h38m48.230634536s) [upload=13037]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:33.297944141 (12h39m7.813046721s) [upload=13046]: Starting upload Aug 7 22:18:33.298041745 (12h39m7.813144331s) [upload=13046]: Done starting upload Aug 7 22:18:33.298133963 (12h39m7.813236527s) [upload=13046]: Uploading segments... Aug 7 22:18:33.298195648 (12h39m7.813298236s) [upload=13046]: Got next segment. Inline: true Aug 7 22:18:33.298201348 (12h39m7.813303934s) [upload=13046]: Waiting for error group managing segments... Aug 7 22:18:33.298250060 (12h39m7.813352642s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:33.298255690 (12h39m7.813358273s) [upload=13046]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:33.298259809 (12h39m7.813362387s) [upload=13046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:03.266409729 (12h39m37.781512306s) [upload=13046]: Done uploading segments... Aug 7 22:19:03.266458971 (12h39m37.781561549s) [upload=13046]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000610907s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXX.X....X.X.................................XXXXXXXXXX] duration: 30.001809192s FAIL (2025-08-07 22:18:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001839899s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XX.X..X..X..........................XXXXXXXXXX] duration: 30.001083063s FAIL (2025-08-07 22:18:49 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00110753s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.X.................................XXXXXXX] duration: 1m0.000756037s FAIL (2025-08-07 22:18:39 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2:08:55.672581371 (12h29m30.187684016s) [upload=12903]: Done starting upload Aug 7 22:08:55.672796946 (12h29m30.187899524s) [upload=12903]: Uploading segments... Aug 7 22:08:55.672988832 (12h29m30.188091398s) [upload=12903]: Got next segment. Inline: true Aug 7 22:08:55.673020300 (12h29m30.188122877s) [upload=12903]: Waiting for error group managing segments... Aug 7 22:08:55.673111302 (12h29m30.188213889s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:55.673117396 (12h29m30.188219973s) [upload=12903]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:55.673121451 (12h29m30.188224017s) [upload=12903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:55.748990886 (12h29m30.264093475s) [upload=12903]: Done uploading segments... Aug 7 22:08:55.749011264 (12h29m30.264113844s) [upload=12903]: Upload finished. err: <nil> Aug 7 22:09:33.459458247 (12h30m7.974560823s) [upload=12913]: Starting upload Aug 7 22:09:33.459560333 (12h30m7.974662911s) [upload=12913]: Done starting upload Aug 7 22:09:33.459666234 (12h30m7.974768811s) [upload=12913]: Uploading segments... Aug 7 22:09:33.459731513 (12h30m7.974834099s) [upload=12913]: Got next segment. Inline: true Aug 7 22:09:33.459737908 (12h30m7.974840491s) [upload=12913]: Waiting for error group managing segments... Aug 7 22:09:33.459810629 (12h30m7.974913195s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:33.459816521 (12h30m7.974919097s) [upload=12913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:33.459819752 (12h30m7.974922328s) [upload=12913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:33.522052292 (12h30m8.037154862s) [upload=12913]: Done uploading segments... Aug 7 22:09:33.522072074 (12h30m8.037174651s) [upload=12913]: Upload finished. err: <nil> Aug 7 22:10:20.772089130 (12h30m55.287191696s) [upload=12928]: Starting upload Aug 7 22:10:20.772161634 (12h30m55.28726421s) [upload=12928]: Done starting upload Aug 7 22:10:20.772231905 (12h30m55.287334472s) [upload=12928]: Uploading segments... Aug 7 22:10:20.772284367 (12h30m55.287386947s) [upload=12928]: Got next segment. Inline: true Aug 7 22:10:20.772289092 (12h30m55.287391658s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:10:20.772344988 (12h30m55.287447554s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:20.772349640 (12h30m55.287452217s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:20.772352965 (12h30m55.287455541s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:20.857003207 (12h30m55.372105787s) [upload=12928]: Done uploading segments... Aug 7 22:10:20.857032212 (12h30m55.372134783s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:11:23.331604570 (12h31m57.846707137s) [upload=12944]: Starting upload Aug 7 22:11:23.331668435 (12h31m57.846771012s) [upload=12944]: Done starting upload Aug 7 22:11:23.331728315 (12h31m57.846830881s) [upload=12944]: Uploading segments... Aug 7 22:11:23.331766773 (12h31m57.846869355s) [upload=12944]: Got next segment. Inline: true Aug 7 22:11:23.331771532 (12h31m57.846874108s) [upload=12944]: Waiting for error group managing segments... Aug 7 22:11:23.331812523 (12h31m57.846915099s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:23.331817222 (12h31m57.846919788s) [upload=12944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:23.331820650 (12h31m57.846923226s) [upload=12944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:23.301325187 (12h32m57.816427773s) [upload=12944]: Done uploading segments... Aug 7 22:12:23.301350364 (12h32m57.816452947s) [upload=12944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:33.831933438 (12h33m8.34703602s) [upload=12965]: Starting upload Aug 7 22:12:33.832018368 (12h33m8.34712093s) [upload=12965]: Done starting upload Aug 7 22:12:33.832086961 (12h33m8.347189539s) [upload=12965]: Uploading segments... Aug 7 22:12:33.832135510 (12h33m8.347238093s) [upload=12965]: Got next segment. Inline: true Aug 7 22:12:33.832141249 (12h33m8.347243826s) [upload=12965]: Waiting for error group managing segments... Aug 7 22:12:33.832194093 (12h33m8.347296674s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:33.832199847 (12h33m8.347302426s) [upload=12965]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:33.832205088 (12h33m8.347307666s) [upload=12965]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:33.802839016 (12h34m8.317941593s) [upload=12965]: Done uploading segments... Aug 7 22:13:33.802913313 (12h34m8.31801589s) [upload=12965]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.913275619 (12h34m8.428378249s) [upload=12977]: Starting upload Aug 7 22:13:33.913398357 (12h34m8.428500941s) [upload=12977]: Done starting upload Aug 7 22:13:33.913496779 (12h34m8.428599348s) [upload=12977]: Uploading segments... Aug 7 22:13:33.913568818 (12h34m8.428671424s) [upload=12977]: Got next segment. Inline: true Aug 7 22:13:33.913576485 (12h34m8.428679072s) [upload=12977]: Waiting for error group managing segments... Aug 7 22:13:33.913646668 (12h34m8.428749234s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.913652247 (12h34m8.428754834s) [upload=12977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.913655814 (12h34m8.428758404s) [upload=12977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.874545239 (12h35m8.389647805s) [upload=12977]: Done uploading segments... Aug 7 22:14:33.874575517 (12h35m8.389678082s) [upload=12977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:38.961368058 (12h35m13.476470627s) [upload=12995]: Starting upload Aug 7 22:14:38.961455294 (12h35m13.476557856s) [upload=12995]: Done starting upload Aug 7 22:14:38.961527233 (12h35m13.476629807s) [upload=12995]: Uploading segments... Aug 7 22:14:38.961574411 (12h35m13.476676997s) [upload=12995]: Got next segment. Inline: true Aug 7 22:14:38.961580546 (12h35m13.47668312s) [upload=12995]: Waiting for error group managing segments... Aug 7 22:14:38.961644878 (12h35m13.476747457s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:38.961650553 (12h35m13.476753133s) [upload=12995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:38.961655154 (12h35m13.476757734s) [upload=12995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:38.926876179 (12h36m13.441978762s) [upload=12995]: Done uploading segments... Aug 7 22:15:38.926898754 (12h36m13.442001334s) [upload=12995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:39.023006560 (12h36m13.538109126s) [upload=13009]: Starting upload Aug 7 22:15:39.023159911 (12h36m13.538262487s) [upload=13009]: Done starting upload Aug 7 22:15:39.023273936 (12h36m13.538376511s) [upload=13009]: Uploading segments... Aug 7 22:15:39.023358183 (12h36m13.538460767s) [upload=13009]: Got next segment. Inline: true Aug 7 22:15:39.023363552 (12h36m13.538466129s) [upload=13009]: Waiting for error group managing segments... Aug 7 22:15:39.023458469 (12h36m13.538561049s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:39.023464439 (12h36m13.538567019s) [upload=13009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:39.023491067 (12h36m13.53859365s) [upload=13009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:38.991189889 (12h37m13.506292477s) [upload=13009]: Done uploading segments... Aug 7 22:16:38.991214272 (12h37m13.506316848s) [upload=13009]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:39.116496595 (12h37m13.631599182s) [upload=13020]: Starting upload Aug 7 22:16:39.116578790 (12h37m13.631681366s) [upload=13020]: Done starting upload Aug 7 22:16:39.116734512 (12h37m13.631837083s) [upload=13020]: Uploading segments... Aug 7 22:16:39.116797388 (12h37m13.631899986s) [upload=13020]: Got next segment. Inline: true Aug 7 22:16:39.116803839 (12h37m13.631906421s) [upload=13020]: Waiting for error group managing segments... Aug 7 22:16:39.116855539 (12h37m13.631958104s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:39.116860739 (12h37m13.631963318s) [upload=13020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:39.116864309 (12h37m13.631966875s) [upload=13020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:39.087205419 (12h38m13.602307996s) [upload=13020]: Done uploading segments... Aug 7 22:17:39.087240451 (12h38m13.602343028s) [upload=13020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:39.195279517 (12h38m13.710382082s) [upload=13033]: Starting upload Aug 7 22:17:39.195345128 (12h38m13.710447708s) [upload=13033]: Done starting upload Aug 7 22:17:39.195408125 (12h38m13.710510701s) [upload=13033]: Uploading segments... Aug 7 22:17:39.195557815 (12h38m13.710660402s) [upload=13033]: Got next segment. Inline: true Aug 7 22:17:39.195566565 (12h38m13.710669149s) [upload=13033]: Waiting for error group managing segments... Aug 7 22:17:39.195637820 (12h38m13.710740397s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:39.195643205 (12h38m13.710745782s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:39.195646438 (12h38m13.710749014s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:39.157680356 (12h39m13.672782934s) [upload=13033]: Done uploading segments... Aug 7 22:18:39.157703138 (12h39m13.672805715s) [upload=13033]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000771003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXX.....X..........................XXXXX] duration: 1m0.001205067s FAIL (2025-08-07 22:18:27 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== RGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.087350241 (12h31m52.602452811s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.088850336 (12h31m52.603952921s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Uploading piece... Aug 7 22:11:18.178629350 (12h31m52.693731915s) [upload=12943]: Next returned nil segment Aug 7 22:11:18.178647680 (12h31m52.693750267s) [upload=12943]: Waiting for error group managing segments... Aug 7 22:11:18.197176897 (12h31m52.712279474s) [noise=true , address=104.36.82.9:4020 , piece_id=DM6JIICVKVDC2EKVDPTLSGBG3D6TZY4CCCQDNIMVHD65D26OWO5Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.201535099 (12h31m52.716637672s) [noise=true , address=104.36.82.3:4791 , piece_id=4DU2IZNC53Y5ILXD4BGJCDAZ4PPPC4VEM7RFCJLPIRSQ4TVVRWHQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.212513462 (12h31m52.727616038s) [noise=true , address=104.36.82.47:4552 , piece_id=NJTH3OJMQQBV6PEADF4IFYZ3U7QZHJIU774CEPVY5BBBXI6V53QA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.213955699 (12h31m52.729058282s) [noise=true , address=104.36.82.11:4146 , piece_id=ECKK26XDQMG6V3CKUIVCIANYBY4PURYRNMFRJTBA77JRLINKMUVQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.214449276 (12h31m52.729551846s) [noise=true , address=104.36.82.44:4890 , piece_id=WM7ZSYZI3MEGKOALRONSD4VPPTDAW6KTTBC4Y7Q5M7IADZAA7VJA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.217895630 (12h31m52.732998217s) [noise=true , address=45.91.116.14:4221 , piece_id=HMZRWRVAWWRVUKLEDZDHTJPMETNONXRGDHCKFBIZ3465EPF45GMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.218622391 (12h31m52.733724974s) [noise=true , address=104.36.82.54:4169 , piece_id=U5IJPZ7U34WBDTUIG43623FOQWAABNCBQNRCEN5SG32JKDMBQ3PA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.219619423 (12h31m52.734722004s) [noise=true , address=45.91.116.5:4320 , piece_id=T7PZIUULEQRKQVMMKJPD6NKQC3AHJ22YC7XCRYOD5UBZ5JTT4QKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.227715032 (12h31m52.742817609s) [noise=true , address=104.36.82.41:4788 , piece_id=OJNZBIBN6K2IYLWIMVMNO7BLY5FKJ3SWVWZRFH3VQPEYUHRCHZIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.228516300 (12h31m52.743618876s) [noise=true , address=45.91.116.8:4061 , piece_id=MA2AHF32LTBEJ5KRGXO2S3EKSAPRNGGS3VDHUN2R74UH6DPDEJIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.241530379 (12h31m52.756632941s) [noise=true , address=104.36.82.5:4859 , piece_id=OS47JVGIPM4HBWV6TPEBYPRLI3NVEEJ4ZPPHBIWSTHEQVNPMFBIQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.251935797 (12h31m52.767038375s) [noise=true , address=104.36.82.37:4668 , piece_id=XRJQ5CBZQZIM6E6SMEOZWU63UMUIW2TBFA7GD7SI3SRA27P243WA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.277091280 (12h31m52.792193849s) [noise=true , address=104.36.82.34:4555 , piece_id=QDJV4U33AJP5SJVS3BSVIOQFDSFOZ5P4XAP3AH3Z525K57U5VOKA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291897936 (12h31m52.807000513s) [noise=true , address=104.36.82.60:4342 , piece_id=5LZ247YTSW7ABTMGS75QYX5ATR5O47XU7UAOSNH2DYHXSE36FN2Q, seg_pos={0 0}, upload=12943]: Done uploading piece. err: <nil> Aug 7 22:11:18.291922152 (12h31m52.80702473s) [seg_pos={0 0}, upload=12943]: Segment reached optimal threshold of 14 pieces. Aug 7 22:11:18.292352474 (12h31m52.807455054s) [noise=true , address=104.36.82.35:4591 , piece_id=KUKTQY554VFN5NOCL7L4W3TJXLSPOZYECLMZGJWNQC26XH5W6P5A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a453d6xSb26epbgqPuco3UPTf9yzoxBeKToCUMNiNHUddAq55): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:18.292749416 (12h31m52.807851996s) [noise=true , address=104.36.82.61:4383 , piece_id=YTMXPHQSLO6AHZCGEA3KR5YUGMHH3UPGEX26JNQ6MXSIKW6AJUMA, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N24q6Li2xavkE61KntNxDPwQzTVHaG4LxyEqFUT1hhGYNQ9mp): protocol: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45544->104.36.82.61:4383: use of closed network connection Aug 7 22:11:18.293007428 (12h31m52.808110005s) [noise=true , address=64.49.13.26:4950 , piece_id=DW4TXDRT7ZGXOOU3RKT4ACLWMYFAGOPD6Q2HV7TFESGXSOC4DE6A, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: read tcp 159.223.143.39:34280->64.49.13.26:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293070269 (12h31m52.808172846s) [noise=true , address=104.36.82.45:4909 , piece_id=UYJL3JPM7VCPGSXZ5J263PRO5K75MLZIWQTA3QCPQEQNXN3LVTNQ, seg_pos={0 0}, upload=12943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: read tcp 159.223.143.39:46438->104.36.82.45:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:18.293086568 (12h31m52.808189151s) [seg_pos={0 0}, upload=12943]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:11:18.293166574 (12h31m52.808269155s) [upload=12943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:18.293172005 (12h31m52.808274588s) [upload=12943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:08.727892812 (12h34m43.242995378s) [upload=12943]: Done uploading segments... Aug 7 22:14:08.727919062 (12h34m43.243021627s) [upload=12943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:08.820786996 (12h34m43.335889564s) [upload=12987]: Starting upload Aug 7 22:14:08.820869466 (12h34m43.335972332s) [upload=12987]: Done starting upload Aug 7 22:14:08.820990812 (12h34m43.336093389s) [upload=12987]: Uploading segments... Aug 7 22:14:08.821180945 (12h34m43.336283523s) [upload=12987]: Got next segment. Inline: false Aug 7 22:14:08.821227735 (12h34m43.336330315s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:08.821233790 (12h34m43.336336478s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:08.788378642 (12h35m43.303481218s) [upload=12987]: Next segment err: context deadline exceeded Aug 7 22:15:08.788567212 (12h35m43.303669789s) [upload=12987]: Done uploading segments... Aug 7 22:15:08.788593806 (12h35m43.303696388s) [upload=12987]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.518494042 (12h35m57.033596616s) [upload=13004]: Starting upload Aug 7 22:15:22.518605977 (12h35m57.033708558s) [upload=13004]: Done starting upload Aug 7 22:15:22.518927572 (12h35m57.034030143s) [upload=13004]: Uploading segments... Aug 7 22:15:22.519083330 (12h35m57.034185915s) [upload=13004]: Got next segment. Inline: false Aug 7 22:15:22.519139330 (12h35m57.034241894s) [upload=13004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.519145674 (12h35m57.034248259s) [upload=13004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.486086322 (12h36m57.001188918s) [upload=13004]: Next segment err: context deadline exceeded Aug 7 22:16:22.486315849 (12h36m57.00141843s) [upload=13004]: Done uploading segments... Aug 7 22:16:22.486353471 (12h36m57.001456038s) [upload=13004]: Upload finished. err: context deadline exceeded Aug 7 22:16:26.977501497 (12h37m1.492604079s) [upload=13017]: Starting upload Aug 7 22:16:26.977634110 (12h37m1.492736693s) [upload=13017]: Done starting upload Aug 7 22:16:26.977724277 (12h37m1.492826848s) [upload=13017]: Uploading segments... Aug 7 22:16:26.978074717 (12h37m1.493177287s) [upload=13017]: Got next segment. Inline: false Aug 7 22:16:26.978159060 (12h37m1.493261638s) [upload=13017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:26.978164291 (12h37m1.493266867s) [upload=13017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:26.948529899 (12h38m1.463632483s) [upload=13017]: Next segment err: context deadline exceeded Aug 7 22:17:26.948553603 (12h38m1.463656182s) [upload=13017]: Done uploading segments... Aug 7 22:17:26.948580927 (12h38m1.463683497s) [upload=13017]: Upload finished. err: context deadline exceeded Aug 7 22:17:27.068460198 (12h38m1.583562778s) [upload=13032]: Starting upload Aug 7 22:17:27.068564049 (12h38m1.583666643s) [upload=13032]: Done starting upload Aug 7 22:17:27.068889168 (12h38m1.583991745s) [upload=13032]: Uploading segments... Aug 7 22:17:27.069219219 (12h38m1.584321786s) [upload=13032]: Got next segment. Inline: false Aug 7 22:17:27.069277883 (12h38m1.584380461s) [upload=13032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:27.069305785 (12h38m1.58440837s) [upload=13032]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:27.039567178 (12h39m1.554669767s) [upload=13032]: Next segment err: context deadline exceeded Aug 7 22:18:27.039666174 (12h39m1.554768736s) [upload=13032]: Done uploading segments... Aug 7 22:18:27.039688215 (12h39m1.554790778s) [upload=13032]: Upload finished. err: context deadline exceeded Duration: 1m0.001224224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXX....XX................................XXXXXXXXXX] duration: 30.00138575s FAIL (2025-08-07 22:18:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001424896s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 288.95082ms FAIL (2025-08-07 22:18:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 288.983466ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXX.X....X.X.................................XXXXXXXXX] duration: 30.000896156s FAIL (2025-08-07 22:18:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000917742s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XX...XXX...............................XXXXXXXXX] duration: 30.000638957s FAIL (2025-08-07 22:18:13 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g upload Aug 7 22:10:18.242410011 (12h30m52.757512577s) [upload=12926]: Uploading segments... Aug 7 22:10:18.242471376 (12h30m52.757573961s) [upload=12926]: Got next segment. Inline: true Aug 7 22:10:18.242478056 (12h30m52.757580635s) [upload=12926]: Waiting for error group managing segments... Aug 7 22:10:18.242535768 (12h30m52.757638347s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:18.242540806 (12h30m52.757643383s) [upload=12926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:18.242544109 (12h30m52.757646685s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:18.309606842 (12h30m52.824709422s) [upload=12926]: Done uploading segments... Aug 7 22:10:18.309625536 (12h30m52.824728119s) [upload=12926]: Upload finished. err: <nil> Aug 7 22:11:28.337393719 (12h32m2.852496289s) [upload=12946]: Starting upload Aug 7 22:11:28.337459886 (12h32m2.852562466s) [upload=12946]: Done starting upload Aug 7 22:11:28.337521996 (12h32m2.852624566s) [upload=12946]: Uploading segments... Aug 7 22:11:28.337560362 (12h32m2.852662948s) [upload=12946]: Got next segment. Inline: true Aug 7 22:11:28.337565059 (12h32m2.852667634s) [upload=12946]: Waiting for error group managing segments... Aug 7 22:11:28.337604389 (12h32m2.852706965s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:28.337608838 (12h32m2.852711414s) [upload=12946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:28.337612338 (12h32m2.852714915s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:58.307179904 (12h32m32.82228248s) [upload=12946]: Done uploading segments... Aug 7 22:11:58.307198958 (12h32m32.822301534s) [upload=12946]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:21.223007637 (12h32m55.738110218s) [upload=12959]: Starting upload Aug 7 22:12:21.223121202 (12h32m55.738223785s) [upload=12959]: Done starting upload Aug 7 22:12:21.223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:19.727995115 (12h36m54.243097692s) [upload=13016]: Starting upload Aug 7 22:16:19.728064428 (12h36m54.243167004s) [upload=13016]: Done starting upload Aug 7 22:16:19.728134488 (12h36m54.243237051s) [upload=13016]: Uploading segments... Aug 7 22:16:19.728182154 (12h36m54.243284736s) [upload=13016]: Got next segment. Inline: true Aug 7 22:16:19.728186380 (12h36m54.243288955s) [upload=13016]: Waiting for error group managing segments... Aug 7 22:16:19.728223320 (12h36m54.243325899s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:19.728227150 (12h36m54.243329727s) [upload=13016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:19.728229527 (12h36m54.243332103s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:49.691229995 (12h37m24.206332575s) [upload=13016]: Done uploading segments... Aug 7 22:16:49.691252411 (12h37m24.206354988s) [upload=13016]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:13.655336229 (12h37m48.170438797s) [upload=13028]: Starting upload Aug 7 22:17:13.655409916 (12h37m48.170512492s) [upload=13028]: Done starting upload Aug 7 22:17:13.655478262 (12h37m48.17058083s) [upload=13028]: Uploading segments... Aug 7 22:17:13.655548537 (12h37m48.17065113s) [upload=13028]: Got next segment. Inline: true Aug 7 22:17:13.655555256 (12h37m48.170657826s) [upload=13028]: Waiting for error group managing segments... Aug 7 22:17:13.655620901 (12h37m48.170723469s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:13.655627928 (12h37m48.170730508s) [upload=13028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:13.655632202 (12h37m48.170734786s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:43.624980277 (12h38m18.140082854s) [upload=13028]: Done uploading segments... Aug 7 22:17:43.624997725 (12h38m18.140100301s) [upload=13028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:43.747114121 (12h38m18.2622167s) [upload=13037]: Starting upload Aug 7 22:17:43.747186113 (12h38m18.262288692s) [upload=13037]: Done starting upload Aug 7 22:17:43.747260024 (12h38m18.262362591s) [upload=13037]: Uploading segments... Aug 7 22:17:43.747409410 (12h38m18.262511987s) [upload=13037]: Got next segment. Inline: true Aug 7 22:17:43.747414935 (12h38m18.262517512s) [upload=13037]: Waiting for error group managing segments... Aug 7 22:17:43.747457457 (12h38m18.262560036s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:43.747461997 (12h38m18.262564572s) [upload=13037]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:43.747465159 (12h38m18.262567736s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:13.715508579 (12h38m48.230611159s) [upload=13037]: Done uploading segments... Aug 7 22:18:13.715531957 (12h38m48.230634536s) [upload=13037]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000657592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........................................X.......] duration: 726.759054ms FAIL (2025-08-07 22:18:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 70b8ffb330ca946ca16cbba98295b25e9d73c23ceaf12561cb0410c721509eb0 != before 7a7c50f9088555b2e513a4e1d17c684f3e108f16da50d5cb1fb6d77de72f8388 ======LOGS====== oad finished. err: <nil> Aug 7 22:12:09.370677362 (12h32m43.885779928s) [upload=12956]: Starting upload Aug 7 22:12:09.370743926 (12h32m43.885846503s) [upload=12956]: Done starting upload Aug 7 22:12:09.371065555 (12h32m43.886168183s) [upload=12956]: Uploading segments... Aug 7 22:12:09.371127350 (12h32m43.886229927s) [upload=12956]: Got next segment. Inline: false Aug 7 22:12:09.371179942 (12h32m43.886282519s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:09.371184578 (12h32m43.886287156s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:09.514484447 (12h32m44.029587032s) [seg_pos={0 0}, upload=12956]: Begin upload segment... Aug 7 22:12:09.514553324 (12h32m44.029655914s) [seg_pos={0 0}, upload=12956]: Done begin upload segment. Aug 7 22:12:09.514604463 (12h32m44.029707051s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.514656287 (12h32m44.029758882s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515221313 (12h32m44.030323899s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515435936 (12h32m44.030538517s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.515609669 (12h32m44.030712245s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516008521 (12h32m44.03111111s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516150437 (12h32m44.031253023s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516344626 (12h32m44.031447208s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.516841797 (12h32m44.031944374s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517033011 (12h32m44.032135594s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517240434 (12h32m44.032343025s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517628325 (12h32m44.032730907s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.517733543 (12h32m44.032836126s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518003213 (12h32m44.033105792s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518348003 (12h32m44.03345058s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.518755006 (12h32m44.033857585s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.522094892 (12h32m44.037197472s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.523039320 (12h32m44.038141897s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Uploading piece... Aug 7 22:12:09.590300484 (12h32m44.105403049s) [upload=12956]: Next returned nil segment Aug 7 22:12:09.590317010 (12h32m44.105419591s) [upload=12956]: Waiting for error group managing segments... Aug 7 22:12:09.604186438 (12h32m44.119289008s) [noise=true , address=45.91.116.11:4132 , piece_id=7DYDCKW5NLR22DRUCWIMELTSHM325ESELDERG26BTA2YLWQE2HHA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.606286681 (12h32m44.121389287s) [noise=true , address=104.36.82.59:4326 , piece_id=SHCRSGVR3LRTNV2GEXSWQWSOYPHUOOHDJPYOUALO4M34TZGCHHBA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620487996 (12h32m44.135590569s) [noise=true , address=104.36.82.61:4386 , piece_id=PNWQU2MPHMRVDXWMMWVNLWGLFQAMEJRLGJXFRCT4M3QYDEG5GSFQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.620594571 (12h32m44.135697142s) [noise=true , address=77.90.31.14:4683 , piece_id=4NNJC6OFHOZNQIKLWRGP7JOCMJAF5NHGTF24A6TID7JVKXNFZATQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.625884604 (12h32m44.140987167s) [noise=true , address=104.36.82.15:4267 , piece_id=DSRR5EHTM75MM7P4MG2YNPHWAE7WHALDCRE4L3YZKKH2BMKRTKAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.626396996 (12h32m44.141499578s) [noise=true , address=104.36.82.58:4292 , piece_id=PFHGP3WHANRR4CSDD3VN5MBXQ65OV7UGJ5WDUP34AZYQZAMRLU7A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.629464012 (12h32m44.144566574s) [noise=true , address=104.36.82.5:4844 , piece_id=GDXNIVT4KT5XFLFAR7B7NMHQDCE2OGGJQYHWID5ASAWO2IVNBN7Q, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643428107 (12h32m44.158530675s) [noise=true , address=104.36.82.14:4248 , piece_id=7ZRBZNDCIDWWONITBOTR3OKLOWVCZ5VCANY6YAQQW6JYSUBUMJZA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.643719221 (12h32m44.15882179s) [noise=true , address=104.36.82.44:4873 , piece_id=GLETUFDVURZ3XPA2OHFTOA7NLKIQSPKF6HVFNPGBOOPU7ICLEOAA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.644290441 (12h32m44.159393003s) [noise=true , address=104.36.82.34:4575 , piece_id=MQE24KHKGXFJPHPKZH4NAI3G2FSOWA42JGN2WEXU47J6ALXEMFOQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.646798488 (12h32m44.161901065s) [noise=true , address=104.36.82.43:4856 , piece_id=PIKKNIOZICWM7EJM3NSRIJWPLBXIAP7WNT7RBREAHU6JRXZDCHYA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.656991689 (12h32m44.172094259s) [noise=true , address=104.36.82.56:4257 , piece_id=6CTH2UDV6ECR5GOBNLG36WD63IHMQTVYH3IUYH33RZYB4ZQ6LQ2A, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:09.835525769 (12h32m44.350628346s) [noise=true , address=79.127.201.215:4357 , piece_id=I2SQGM7NHIUHKNQG6RT3PH3W6P5OKJPP3MFJ65QC7CTN36DUYYJA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167762322 (12h32m44.682864893s) [noise=true , address=64.49.13.154:4934 , piece_id=5T4YXQHU4DJEZCEAHQT3U4P636W5WDGSQCVCB73XQOCJFAV4O2MA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: <nil> Aug 7 22:12:10.167785966 (12h32m44.682888542s) [seg_pos={0 0}, upload=12956]: Segment reached optimal threshold of 14 pieces. Aug 7 22:12:10.167971590 (12h32m44.683074167s) [noise=true , address=64.49.13.137:4493 , piece_id=432XMLABWE3DSWXU3B3BTREMG6CP7ILTNU4RJPA6JLHEJNX5POIA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N91KV4t7yYJvKLFMaXHjByCZpsNgWFnYyZHJE7BqEYBGMLAi9): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168073316 (12h32m44.683175878s) [noise=true , address=104.36.82.18:4383 , piece_id=XVON7B5A6KIUDIZUJKNMNDQBQ5JNTCORJO7JL4NLPYOIGYU63TXQ, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168146785 (12h32m44.683249347s) [noise=true , address=64.49.13.138:4490 , piece_id=RT5AQYMACT6OGCNBPVOJOLYGYQHHND6TVS3FEE52MXZWXZJLCCVA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Jp1vcm8dpppQHEfAtYmM29CDS3mqhojhuVrzkL3PsYHW2q8zA): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168258506 (12h32m44.683361075s) [noise=true , address=79.127.205.235:4670 , piece_id=3GWZPOLPZFZUAJKZ6MOXRTGGE7YDBMNP53WLJXEVOYW2ES5MLZPA, seg_pos={0 0}, upload=12956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JgTfMzsd3rMqQwNim6BoZSiePoLW3gcUtraj84Kn4Szy7aNga): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:10.168271822 (12h32m44.68337439s) [seg_pos={0 0}, upload=12956]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:12:10.168367951 (12h32m44.683470527s) [upload=12956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:10.168372753 (12h32m44.683475329s) [upload=12956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:11.355196489 (12h32m45.870299059s) [upload=12956]: Done uploading segments... Aug 7 22:12:11.355214126 (12h32m45.870316692s) [upload=12956]: Upload finished. err: <nil> Duration: 726.791719ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XX...X.............................XXXXXXX] duration: 1m0.001169961s FAIL (2025-08-07 22:18:06 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001215816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XX.X..X..X..........................XXXXXXXXX] duration: 30.000704857s FAIL (2025-08-07 22:17:45 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000720123s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XX...XXX...............................XXXXXXXX] duration: 30.000483104s FAIL (2025-08-07 22:17:43 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9s) [upload=12914]: Done starting upload Aug 7 22:09:35.598909885 (12h30m10.11401245s) [upload=12914]: Uploading segments... Aug 7 22:09:35.598948468 (12h30m10.114051051s) [upload=12914]: Got next segment. Inline: true Aug 7 22:09:35.598952824 (12h30m10.114055403s) [upload=12914]: Waiting for error group managing segments... Aug 7 22:09:35.599006219 (12h30m10.114108795s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:35.599011436 (12h30m10.114114015s) [upload=12914]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:35.599014692 (12h30m10.114117272s) [upload=12914]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:35.668939111 (12h30m10.184041679s) [upload=12914]: Done uploading segments... Aug 7 22:09:35.668959039 (12h30m10.184061615s) [upload=12914]: Upload finished. err: <nil> Aug 7 22:10:18.242162305 (12h30m52.757264891s) [upload=12926]: Starting upload Aug 7 22:10:18.242275276 (12h30m52.75737786s) [upload=12926]: Done starting upload Aug 7 22:10:18.242410011 (12h30m52.757512577s) [upload=12926]: Uploading segments... Aug 7 22:10:18.242471376 (12h30m52.757573961s) [upload=12926]: Got next segment. Inline: true Aug 7 22:10:18.242478056 (12h30m52.757580635s) [upload=12926]: Waiting for error group managing segments... Aug 7 22:10:18.242535768 (12h30m52.757638347s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:18.242540806 (12h30m52.757643383s) [upload=12926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:18.242544109 (12h30m52.757646685s) [upload=12926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:18.309606842 (12h30m52.824709422s) [upload=12926]: Done uploading segments... Aug 7 22:10:18.309625536 (12h30m52.824728119s) [upload=12926]: Upload finished. err: <nil> Aug 7 22:11:28.337393719 (12h32m2.852496289s) [upload=12946]: Starting upload Aug 7 22:11:28.337459886 (12h32m2.852562466s) [upload=12946]: Done starting upload Aug 7 22:11:28.337521996 (12h32m2.852624566s) [upload=12946]: Uploading segments... Aug 7 22:11:28.337560362 (12h32m2.852662948s) [upload=12946]: Got next segment. Inline: true Aug 7 22:11:28.337565059 (12h32m2.852667634s) [upload=12946]: Waiting for error group managing segments... Aug 7 22:11:28.337604389 (12h32m2.852706965s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:28.337608838 (12h32m2.852711414s) [upload=12946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:28.337612338 (12h32m2.852714915s) [upload=12946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:58.307179904 (12h32m32.82228248s) [upload=12946]: Done uploading segments... Aug 7 22:11:58.307198958 (12h32m32.822301534s) [upload=12946]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:21.223007637 (12h32m55.738110218s) [upload=12959]: Starting upload Aug 7 22:12:21.223121202 (12h32m55.738223785s) [upload=12959]: Done starting upload Aug 7 22:12:21.223195673 (12h32m55.738298237s) [upload=12959]: Uploading segments... Aug 7 22:12:21.223245128 (12h32m55.738347714s) [upload=12959]: Got next segment. Inline: true Aug 7 22:12:21.223251709 (12h32m55.738354294s) [upload=12959]: Waiting for error group managing segments... Aug 7 22:12:21.223301612 (12h32m55.73840419s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:21.223307605 (12h32m55.73841019s) [upload=12959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:21.223313220 (12h32m55.738415805s) [upload=12959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:51.190833766 (12h33m25.705936354s) [upload=12959]: Done uploading segments... Aug 7 22:12:51.190855934 (12h33m25.705958518s) [upload=12959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:33.961947992 (12h34m8.477050562s) [upload=12978]: Starting upload Aug 7 22:13:33.962022765 (12h34m8.477125334s) [upload=12978]: Done starting upload Aug 7 22:13:33.962138333 (12h34m8.477240898s) [upload=12978]: Uploading segments... Aug 7 22:13:33.962206972 (12h34m8.47730955s) [upload=12978]: Got next segment. Inline: true Aug 7 22:13:33.962214161 (12h34m8.477316742s) [upload=12978]: Waiting for error group managing segments... Aug 7 22:13:33.962309806 (12h34m8.477412392s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:33.962316375 (12h34m8.477418959s) [upload=12978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:33.962319934 (12h34m8.477422517s) [upload=12978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:03.929089530 (12h34m38.444192112s) [upload=12978]: Done uploading segments... Aug 7 22:14:03.929116637 (12h34m38.444219217s) [upload=12978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.043639757 (12h34m38.558742332s) [upload=12985]: Starting upload Aug 7 22:14:04.043726081 (12h34m38.558828656s) [upload=12985]: Done starting upload Aug 7 22:14:04.043808822 (12h34m38.558911399s) [upload=12985]: Uploading segments... Aug 7 22:14:04.043864623 (12h34m38.558967204s) [upload=12985]: Got next segment. Inline: true Aug 7 22:14:04.043869731 (12h34m38.558972307s) [upload=12985]: Waiting for error group managing segments... Aug 7 22:14:04.043934849 (12h34m38.559037426s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.043942633 (12h34m38.55904521s) [upload=12985]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.043945887 (12h34m38.559048463s) [upload=12985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:34.013311700 (12h35m8.528414277s) [upload=12985]: Done uploading segments... Aug 7 22:14:34.013330957 (12h35m8.528433534s) [upload=12985]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:05.325819723 (12h35m39.840922304s) [upload=12999]: Starting upload Aug 7 22:15:05.325904159 (12h35m39.841006746s) [upload=12999]: Done starting upload Aug 7 22:15:05.325980298 (12h35m39.84108289s) [upload=12999]: Uploading segments... Aug 7 22:15:05.326029165 (12h35m39.841131756s) [upload=12999]: Got next segment. Inline: true Aug 7 22:15:05.326035056 (12h35m39.841137641s) [upload=12999]: Waiting for error group managing segments... Aug 7 22:15:05.326082977 (12h35m39.841185565s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:05.326088149 (12h35m39.84119073s) [upload=12999]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:05.326092128 (12h35m39.841194709s) [upload=12999]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:35.288775559 (12h36m9.803878146s) [upload=12999]: Done uploading segments... Aug 7 22:15:35.288800975 (12h36m9.803903553s) [upload=12999]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:40.665367913 (12h36m15.180470479s) [upload=13010]: Starting upload Aug 7 22:15:40.665434268 (12h36m15.180536845s) [upload=13010]: Done starting upload Aug 7 22:15:40.665499877 (12h36m15.180602442s) [upload=13010]: Uploading segments... Aug 7 22:15:40.665590817 (12h36m15.180693396s) [upload=13010]: Got next segment. Inline: true Aug 7 22:15:40.665597143 (12h36m15.18069972s) [upload=13010]: Waiting for error group managing segments... Aug 7 22:15:40.665671744 (12h36m15.180774307s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:40.665678967 (12h36m15.180781528s) [upload=13010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:40.665681447 (12h36m15.180784023s) [upload=13010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:10.636048401 (12h36m45.151150968s) [upload=13010]: Done uploading segments... Aug 7 22:16:10.636067176 (12h36m45.151169742s) [upload=13010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:19.727995115 (12h36m54.243097692s) [upload=13016]: Starting upload Aug 7 22:16:19.728064428 (12h36m54.243167004s) [upload=13016]: Done starting upload Aug 7 22:16:19.728134488 (12h36m54.243237051s) [upload=13016]: Uploading segments... Aug 7 22:16:19.728182154 (12h36m54.243284736s) [upload=13016]: Got next segment. Inline: true Aug 7 22:16:19.728186380 (12h36m54.243288955s) [upload=13016]: Waiting for error group managing segments... Aug 7 22:16:19.728223320 (12h36m54.243325899s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:19.728227150 (12h36m54.243329727s) [upload=13016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:19.728229527 (12h36m54.243332103s) [upload=13016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:49.691229995 (12h37m24.206332575s) [upload=13016]: Done uploading segments... Aug 7 22:16:49.691252411 (12h37m24.206354988s) [upload=13016]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:13.655336229 (12h37m48.170438797s) [upload=13028]: Starting upload Aug 7 22:17:13.655409916 (12h37m48.170512492s) [upload=13028]: Done starting upload Aug 7 22:17:13.655478262 (12h37m48.17058083s) [upload=13028]: Uploading segments... Aug 7 22:17:13.655548537 (12h37m48.17065113s) [upload=13028]: Got next segment. Inline: true Aug 7 22:17:13.655555256 (12h37m48.170657826s) [upload=13028]: Waiting for error group managing segments... Aug 7 22:17:13.655620901 (12h37m48.170723469s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:13.655627928 (12h37m48.170730508s) [upload=13028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:13.655632202 (12h37m48.170734786s) [upload=13028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:43.624980277 (12h38m18.140082854s) [upload=13028]: Done uploading segments... Aug 7 22:17:43.624997725 (12h38m18.140100301s) [upload=13028]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000507463s core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)