Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXX.XX.X..X..................] duration: 1.918685268s FAIL (2025-08-07 22:01:14 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.918714687s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 205.968357ms FAIL (2025-08-07 22:01:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 205.995518ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.X..................] duration: 709.764217ms FAIL (2025-08-07 22:00:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 709.785491ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.64961122s FAIL (2025-08-07 22:00:53 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 1.649638657s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...X.................] duration: 745.292343ms FAIL (2025-08-07 22:00:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 745.312203ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..X..X...............] duration: 281.045313ms FAIL (2025-08-07 22:00:51 UTC) operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 281.068898ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXX.....X................] duration: 1.361167766s FAIL (2025-08-07 22:00:49 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 68549s) [upload=12537]: Done starting upload Aug 7 21:46:22.615396117 (12h6m57.130498679s) [upload=12537]: Uploading segments... Aug 7 21:46:22.615846476 (12h6m57.13094906s) [upload=12537]: Got next segment. Inline: false Aug 7 21:46:22.615968068 (12h6m57.131070645s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:22.615975064 (12h6m57.131077646s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:28.156865165 (12h7m2.671967741s) [seg_pos={0 0}, upload=12537]: Begin upload segment... Aug 7 21:46:28.156917513 (12h7m2.672020096s) [seg_pos={0 0}, upload=12537]: Done begin upload segment. Aug 7 21:46:28.156967463 (12h7m2.672070048s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157000064 (12h7m2.672102649s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157067642 (12h7m2.672170211s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157406065 (12h7m2.672508645s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157549310 (12h7m2.672651889s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157739828 (12h7m2.672842407s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157989895 (12h7m2.673092473s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158099904 (12h7m2.673202487s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158381871 (12h7m2.673484448s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158490401 (12h7m2.673592979s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158794943 (12h7m2.67389751s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158855471 (12h7m2.673958052s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159214922 (12h7m2.674317506s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159509642 (12h7m2.674612208s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161192583 (12h7m2.676295161s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161765903 (12h7m2.676868473s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162066100 (12h7m2.677168688s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162703851 (12h7m2.67780643s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.268596153 (12h7m2.783698719s) [upload=12537]: Next returned nil segment Aug 7 21:46:28.268617977 (12h7m2.783720553s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:46:28.279079135 (12h7m2.794181715s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.290690859 (12h7m2.805793441s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.292185740 (12h7m2.807288316s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.294418015 (12h7m2.80952059s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.295076589 (12h7m2.810179159s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.297491713 (12h7m2.812594291s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.303643978 (12h7m2.818746547s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.304764566 (12h7m2.819867135s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.315655209 (12h7m2.830757788s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.323593569 (12h7m2.838696138s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.324982970 (12h7m2.840085548s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325108866 (12h7m2.840211445s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325989884 (12h7m2.841092452s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361102046 (12h7m2.876204623s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361164635 (12h7m2.876267211s) [seg_pos={0 0}, upload=12537]: Segment reached optimal threshold of 14 pieces. Aug 7 21:46:28.361368094 (12h7m2.876470663s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361578723 (12h7m2.876681288s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eeQieeLS35wwm6vk8rwcTDPPQgYD9vVhVw8WRt84bduXbaW2j): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361874485 (12h7m2.876977051s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o): protocol: read tcp 159.223.143.39:41094->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361908521 (12h7m2.877011091s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d): protocol: read tcp 159.223.143.39:38696->104.36.82.38:4693: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361930237 (12h7m2.877032803s) [seg_pos={0 0}, upload=12537]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:46:28.362049899 (12h7m2.877152488s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:28.362055767 (12h7m2.877158344s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:22.628005735 (12h7m57.143108314s) [upload=12537]: Done uploading segments... Aug 7 21:47:22.628037085 (12h7m57.143139659s) [upload=12537]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.361219026s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [..............X.........XX..................................] duration: 49.939879ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:00:49 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 49.984136ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....XX..................] duration: 472.192924ms FAIL (2025-08-07 22:00:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.656689ms edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [...........XXX........XX..XX................................] duration: 76.378118ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:00:37 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 76.405666ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.X.................] duration: 519.965885ms FAIL (2025-08-07 22:00:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 519.986062ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 532.466119ms FAIL (2025-08-07 22:00:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 532.502921ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...XXX...................] duration: 286.752696ms FAIL (2025-08-07 22:00:28 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:38.822741159 (12h4m13.337843727s) [upload=12496]: Starting upload Aug 7 21:43:38.822791612 (12h4m13.337894198s) [upload=12496]: Done starting upload Aug 7 21:43:38.822841316 (12h4m13.337943884s) [upload=12496]: Uploading segments... Aug 7 21:43:38.822878518 (12h4m13.337981103s) [upload=12496]: Got next segment. Inline: true Aug 7 21:43:38.822883293 (12h4m13.337985869s) [upload=12496]: Waiting for error group managing segments... Aug 7 21:43:38.822922170 (12h4m13.338024807s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:38.822926518 (12h4m13.338029093s) [upload=12496]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:38.822929823 (12h4m13.338032399s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:38.880695407 (12h4m13.395797984s) [upload=12496]: Done uploading segments... Aug 7 21:43:38.880716578 (12h4m13.395819144s) [upload=12496]: Upload finished. err: <nil> Aug 7 21:44:03.077928978 (12h4m37.593031579s) [upload=12501]: Starting upload Aug 7 21:44:03.078032002 (12h4m37.593134589s) [upload=12501]: Done starting upload Aug 7 21:44:03.078182274 (12h4m37.593284842s) [upload=12501]: Uploading segments... Aug 7 21:44:03.078254104 (12h4m37.59335669s) [upload=12501]: Got next segment. Inline: true Aug 7 21:44:03.078260761 (12h4m37.593363344s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:44:03.078324988 (12h4m37.593427564s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:03.078329732 (12h4m37.593432308s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:03.078333203 (12h4m37.593435779s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:03.150588344 (12h4m37.665690921s) [upload=12501]: Done uploading segments... Aug 7 21:44:03.150609592 (12h4m37.665712177s) [upload=12501]: Upload finished. err: <nil> Aug 7 21:44:56.698719656 (12h5m31.213822219s) [upload=12513]: Starting upload Aug 7 21:44:56.698786775 (12h5m31.213889351s) [upload=12513]: Done starting upload Aug 7 21:44:56.698856755 (12h5m31.213959317s) [upload=12513]: Uploading segments... Aug 7 21:44:56.698908173 (12h5m31.214010758s) [upload=12513]: Got next segment. Inline: true Aug 7 21:44:56.698912338 (12h5m31.214014914s) [upload=12513]: Waiting for error group managing segments... Aug 7 21:44:56.698948274 (12h5m31.21405085s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:56.698965173 (12h5m31.214067748s) [upload=12513]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:56.698967987 (12h5m31.214070563s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:56.781273042 (12h5m31.296375616s) [upload=12513]: Done uploading segments... Aug 7 21:44:56.781296311 (12h5m31.296398893s) [upload=12513]: Upload finished. err: <nil> Aug 7 21:46:10.925701802 (12h6m45.44080437s) [upload=12532]: Starting upload Aug 7 21:46:10.925757954 (12h6m45.440860524s) [upload=12532]: Done starting upload Aug 7 21:46:10.925819534 (12h6m45.440922115s) [upload=12532]: Uploading segments... Aug 7 21:46:10.925869829 (12h6m45.44097242s) [upload=12532]: Got next segment. Inline: true Aug 7 21:46:10.925876475 (12h6m45.440979053s) [upload=12532]: Waiting for error group managing segments... Aug 7 21:46:10.925925157 (12h6m45.441027735s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:10.925930047 (12h6m45.441032624s) [upload=12532]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:10.925933179 (12h6m45.441035756s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:40.893959940 (12h7m15.409062507s) [upload=12532]: Done uploading segments... Aug 7 21:46:40.894015791 (12h7m15.409118368s) [upload=12532]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:17.091619874 (12h7m51.606722441s) [upload=12546]: Starting upload Aug 7 21:47:17.091714186 (12h7m51.606816768s) [upload=12546]: Done starting upload Aug 7 21:47:17.091785367 (12h7m51.606887938s) [upload=12546]: Uploading segments... Aug 7 21:47:17.091863768 (12h7m51.606966354s) [upload=12546]: Got next segment. Inline: true Aug 7 21:47:17.091870088 (12h7m51.606972665s) [upload=12546]: Waiting for error group managing segments... Aug 7 21:47:17.091926672 (12h7m51.607029237s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:17.091932153 (12h7m51.607034729s) [upload=12546]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:17.091935528 (12h7m51.607038105s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:47.062137982 (12h8m21.577240549s) [upload=12546]: Done uploading segments... Aug 7 21:47:47.062157139 (12h8m21.577259705s) [upload=12546]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:55.978154740 (12h8m30.493257315s) [upload=12560]: Starting upload Aug 7 21:47:55.978228943 (12h8m30.493331529s) [upload=12560]: Done starting upload Aug 7 21:47:55.978311008 (12h8m30.493413589s) [upload=12560]: Uploading segments... Aug 7 21:47:55.978374285 (12h8m30.493476868s) [upload=12560]: Got next segment. Inline: true Aug 7 21:47:55.978378891 (12h8m30.493481468s) [upload=12560]: Waiting for error group managing segments... Aug 7 21:47:55.978431682 (12h8m30.493534258s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:55.978436762 (12h8m30.493539338s) [upload=12560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:55.978439848 (12h8m30.493542423s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:25.945081513 (12h9m0.460184093s) [upload=12560]: Done uploading segments... Aug 7 21:48:25.945103520 (12h9m0.460206103s) [upload=12560]: Upload finished. err: metaclient: context deadline exceeded Duration: 286.785518ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....XX.................] duration: 274.556927ms FAIL (2025-08-07 22:00:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 274.583912ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [......X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.............] duration: 2.049982844s FAIL (2025-08-07 22:00:17 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=12497]: Got next segment. Inline: false Aug 7 21:43:52.199899367 (12h4m26.715001952s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:52.199906793 (12h4m26.715009476s) [upload=12497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:22.166742883 (12h4m56.681845464s) [upload=12497]: Next segment err: context deadline exceeded Aug 7 21:44:22.166864816 (12h4m56.681967396s) [upload=12497]: Done uploading segments... Aug 7 21:44:22.166885734 (12h4m56.681988313s) [upload=12497]: Upload finished. err: context deadline exceeded Aug 7 21:45:03.330540016 (12h5m37.845642605s) [upload=12517]: Starting upload Aug 7 21:45:03.330650460 (12h5m37.845753069s) [upload=12517]: Done starting upload Aug 7 21:45:03.330858182 (12h5m37.845960762s) [upload=12517]: Uploading segments... Aug 7 21:45:03.331156974 (12h5m37.84625954s) [upload=12517]: Got next segment. Inline: false Aug 7 21:45:03.331235679 (12h5m37.846338242s) [upload=12517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:03.331241699 (12h5m37.846344282s) [upload=12517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:08.003810167 (12h5m42.518912754s) [seg_pos={0 0}, upload=12517]: Begin upload segment... Aug 7 21:45:08.003882230 (12h5m42.518984812s) [seg_pos={0 0}, upload=12517]: Done begin upload segment. Aug 7 21:45:08.003954509 (12h5m42.519057091s) [noise=true , address=77.90.31.13:4627 , piece_id=O4IROOSYED54KYRJPP6WSHWAGCMW64H2CDMOCH7EGDIBP7W7QVFQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.004263642 (12h5m42.51936623s) [noise=true , address=77.90.31.11:4555 , piece_id=Q3LP2RJ7MFDJABSDGNGWNYB27J2IORSCMA5BIFWZGAOSXF5A5BNQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.004792666 (12h5m42.519895256s) [noise=true , address=45.91.116.10:4083 , piece_id=LBBGLOISIFUGNPBVDLT4MZ5TFC4NTBDC6P2VT5IK6T75IIWEHWLA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.005354037 (12h5m42.520456625s) [noise=true , address=104.36.82.12:4149 , piece_id=57QOP6W7OHKTXBELK3LFIOFFMXD2TWMYSIWLI54URQRPNBLEJ5LA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.005841366 (12h5m42.520943955s) [noise=true , address=104.36.82.8:4969 , piece_id=BSBAVTYONQSVQBD2UQDDXZABBR7TG2TO6ADNCMUDM5A4IZV45UWA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.006328034 (12h5m42.521430625s) [noise=true , address=104.36.82.9:4007 , piece_id=WROTDT7DJG2HGV5UWALZRIZMSGMTH3N7SZ4F7SEI64GSIRLTU2LQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.006815458 (12h5m42.521918046s) [noise=true , address=104.36.82.6:4875 , piece_id=LPFAIF7C6W6YFDM5CHKFHSKWU6EMU4YWW7CSVZZRHD5GCM5BPALA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007331300 (12h5m42.522433909s) [noise=true , address=45.91.116.6:4975 , piece_id=5GDO73EDLMXIVE3OD7FDRAD3LEM5TCALGYSMB5QZNGYSN4I2SCSQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007697466 (12h5m42.522800074s) [noise=true , address=45.91.116.9:4263 , piece_id=64YKT5YSI7RURYDHZGB6SJIHJ6MBXWGE7UVIH6JGVA4XN4HT4XPQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007834257 (12h5m42.522936846s) [noise=true , address=77.90.31.14:4662 , piece_id=NOKVL5CJRIDTPCYOXQCITJO3KRXURIHXTUYEKSLEDWIOV3J6MSCQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.008267290 (12h5m42.523369892s) [noise=true , address=104.36.82.7:4912 , piece_id=TT3R2ZW7RBCVNZJ4YILZROCLUSR5IH6VUUQQCGR5G4O3GQC4JN6A, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.011188107 (12h5m42.52629075s) [noise=true , address=45.91.116.5:4297 , piece_id=MEGO3DATMLAO6UM3V7DPFXWPHVAQWS5I772LC2XQR5G7T6QRPWRA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.012274365 (12h5m42.527376954s) [noise=true , address=45.91.116.10:4082 , piece_id=Z6GB3XGMTCDAQMK2QZWJOGBLAUKLYVOLS6UGRWF4RLY37GVQVRBA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.012957723 (12h5m42.528060311s) [noise=true , address=104.36.82.4:4804 , piece_id=A5LKRZHVGBGBJTAJHJDDG54VOD6F5V2I7A3SWN24YR26JAPTX6WA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.014736176 (12h5m42.529838763s) [noise=true , address=77.90.31.11:4553 , piece_id=567MLMX4HQYW7QN5XNITKR47LJY56IPYJ6GLZYBYPILFYXLPR2JQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.016257825 (12h5m42.531360412s) [noise=true , address=104.36.82.10:4078 , piece_id=HI5SU6YHHN2OQF672MSMGY6RPETV5TCFHGJKI5AGN7NCZO47OXXQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.117745239 (12h5m42.632847808s) [upload=12517]: Next returned nil segment Aug 7 21:45:08.117761018 (12h5m42.632863594s) [upload=12517]: Waiting for error group managing segments... Aug 7 21:45:08.129666998 (12h5m42.644769574s) [noise=true , address=45.91.116.10:4082 , piece_id=Z6GB3XGMTCDAQMK2QZWJOGBLAUKLYVOLS6UGRWF4RLY37GVQVRBA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.130196788 (12h5m42.645299359s) [noise=true , address=77.90.31.11:4555 , piece_id=Q3LP2RJ7MFDJABSDGNGWNYB27J2IORSCMA5BIFWZGAOSXF5A5BNQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.130727325 (12h5m42.645829895s) [noise=true , address=77.90.31.11:4553 , piece_id=567MLMX4HQYW7QN5XNITKR47LJY56IPYJ6GLZYBYPILFYXLPR2JQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.132728107 (12h5m42.647830677s) [noise=true , address=77.90.31.14:4662 , piece_id=NOKVL5CJRIDTPCYOXQCITJO3KRXURIHXTUYEKSLEDWIOV3J6MSCQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.138647804 (12h5m42.653750375s) [noise=true , address=104.36.82.9:4007 , piece_id=WROTDT7DJG2HGV5UWALZRIZMSGMTH3N7SZ4F7SEI64GSIRLTU2LQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.141221377 (12h5m42.656323943s) [noise=true , address=104.36.82.12:4149 , piece_id=57QOP6W7OHKTXBELK3LFIOFFMXD2TWMYSIWLI54URQRPNBLEJ5LA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.142848205 (12h5m42.657950782s) [noise=true , address=45.91.116.5:4297 , piece_id=MEGO3DATMLAO6UM3V7DPFXWPHVAQWS5I772LC2XQR5G7T6QRPWRA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.145051855 (12h5m42.66015442s) [noise=true , address=45.91.116.9:4263 , piece_id=64YKT5YSI7RURYDHZGB6SJIHJ6MBXWGE7UVIH6JGVA4XN4HT4XPQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.145420726 (12h5m42.660523304s) [noise=true , address=45.91.116.6:4975 , piece_id=5GDO73EDLMXIVE3OD7FDRAD3LEM5TCALGYSMB5QZNGYSN4I2SCSQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.146452857 (12h5m42.661555425s) [noise=true , address=104.36.82.8:4969 , piece_id=BSBAVTYONQSVQBD2UQDDXZABBR7TG2TO6ADNCMUDM5A4IZV45UWA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.147080237 (12h5m42.6621828s) [noise=true , address=77.90.31.13:4627 , piece_id=O4IROOSYED54KYRJPP6WSHWAGCMW64H2CDMOCH7EGDIBP7W7QVFQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.148870189 (12h5m42.663972756s) [noise=true , address=104.36.82.4:4804 , piece_id=A5LKRZHVGBGBJTAJHJDDG54VOD6F5V2I7A3SWN24YR26JAPTX6WA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.151158066 (12h5m42.666260643s) [noise=true , address=45.91.116.10:4083 , piece_id=LBBGLOISIFUGNPBVDLT4MZ5TFC4NTBDC6P2VT5IK6T75IIWEHWLA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.152266060 (12h5m42.667368644s) [seg_pos={0 0}, upload=12517]: Segment reached optimal threshold of 13 pieces. Aug 7 21:45:08.152853862 (12h5m42.667956424s) [noise=true , address=104.36.82.7:4912 , piece_id=TT3R2ZW7RBCVNZJ4YILZROCLUSR5IH6VUUQQCGR5G4O3GQC4JN6A, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection; write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection Aug 7 21:45:08.153127403 (12h5m42.668229969s) [noise=true , address=104.36.82.6:4875 , piece_id=LPFAIF7C6W6YFDM5CHKFHSKWU6EMU4YWW7CSVZZRHD5GCM5BPALA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection; write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection Aug 7 21:45:08.154643012 (12h5m42.669745579s) [noise=true , address=104.36.82.10:4078 , piece_id=HI5SU6YHHN2OQF672MSMGY6RPETV5TCFHGJKI5AGN7NCZO47OXXQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 21:45:08.154667840 (12h5m42.669770416s) [seg_pos={0 0}, upload=12517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:45:08.154819614 (12h5m42.669922193s) [upload=12517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:45:08.154825838 (12h5m42.66992865s) [upload=12517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:00.396543819 (12h10m34.911646387s) [upload=12517]: Done uploading segments... Aug 7 21:50:00.396643233 (12h10m34.911745799s) [upload=12517]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.050010632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXX.XX.X..X.................] duration: 1.931955857s FAIL (2025-08-07 22:00:17 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.932011952s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 395.690178ms FAIL (2025-08-07 22:00:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 395.73674ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 30 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...................] duration: 389.731892ms FAIL (2025-08-07 22:00:14 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:02.683404470 (12h4m37.198507059s) [upload=12500]: Starting upload Aug 7 21:44:02.683498983 (12h4m37.198601564s) [upload=12500]: Done starting upload Aug 7 21:44:02.683587655 (12h4m37.198690226s) [upload=12500]: Uploading segments... Aug 7 21:44:02.683635288 (12h4m37.198737882s) [upload=12500]: Got next segment. Inline: true Aug 7 21:44:02.683641097 (12h4m37.198743679s) [upload=12500]: Waiting for error group managing segments... Aug 7 21:44:02.683702304 (12h4m37.198804891s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:02.683709172 (12h4m37.198811757s) [upload=12500]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:02.683713383 (12h4m37.198815961s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.749796670 (12h4m37.264899247s) [upload=12500]: Done uploading segments... Aug 7 21:44:02.749821074 (12h4m37.264923648s) [upload=12500]: Upload finished. err: <nil> Aug 7 21:44:59.056137042 (12h5m33.571239616s) [upload=12515]: Starting upload Aug 7 21:44:59.056204060 (12h5m33.571306636s) [upload=12515]: Done starting upload Aug 7 21:44:59.056275553 (12h5m33.571378122s) [upload=12515]: Uploading segments... Aug 7 21:44:59.056357038 (12h5m33.571459625s) [upload=12515]: Got next segment. Inline: true Aug 7 21:44:59.056364363 (12h5m33.57146694s) [upload=12515]: Waiting for error group managing segments... Aug 7 21:44:59.056439881 (12h5m33.571542451s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:59.056447251 (12h5m33.571549833s) [upload=12515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:59.056453265 (12h5m33.571555847s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:59.018996840 (12h6m33.534099417s) [upload=12515]: Done uploading segments... Aug 7 21:45:59.019015646 (12h6m33.534118222s) [upload=12515]: Upload finished. err: metaclient: context deadline exceeded Duration: 389.765806ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..X..X..............] duration: 438.888452ms FAIL (2025-08-07 22:00:09 UTC) operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 438.96576ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...XXX..................] duration: 253.257411ms FAIL (2025-08-07 22:00:01 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:38.822741159 (12h4m13.337843727s) [upload=12496]: Starting upload Aug 7 21:43:38.822791612 (12h4m13.337894198s) [upload=12496]: Done starting upload Aug 7 21:43:38.822841316 (12h4m13.337943884s) [upload=12496]: Uploading segments... Aug 7 21:43:38.822878518 (12h4m13.337981103s) [upload=12496]: Got next segment. Inline: true Aug 7 21:43:38.822883293 (12h4m13.337985869s) [upload=12496]: Waiting for error group managing segments... Aug 7 21:43:38.822922170 (12h4m13.338024807s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:38.822926518 (12h4m13.338029093s) [upload=12496]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:38.822929823 (12h4m13.338032399s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:38.880695407 (12h4m13.395797984s) [upload=12496]: Done uploading segments... Aug 7 21:43:38.880716578 (12h4m13.395819144s) [upload=12496]: Upload finished. err: <nil> Aug 7 21:44:03.077928978 (12h4m37.593031579s) [upload=12501]: Starting upload Aug 7 21:44:03.078032002 (12h4m37.593134589s) [upload=12501]: Done starting upload Aug 7 21:44:03.078182274 (12h4m37.593284842s) [upload=12501]: Uploading segments... Aug 7 21:44:03.078254104 (12h4m37.59335669s) [upload=12501]: Got next segment. Inline: true Aug 7 21:44:03.078260761 (12h4m37.593363344s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:44:03.078324988 (12h4m37.593427564s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:03.078329732 (12h4m37.593432308s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:03.078333203 (12h4m37.593435779s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:03.150588344 (12h4m37.665690921s) [upload=12501]: Done uploading segments... Aug 7 21:44:03.150609592 (12h4m37.665712177s) [upload=12501]: Upload finished. err: <nil> Aug 7 21:44:56.698719656 (12h5m31.213822219s) [upload=12513]: Starting upload Aug 7 21:44:56.698786775 (12h5m31.213889351s) [upload=12513]: Done starting upload Aug 7 21:44:56.698856755 (12h5m31.213959317s) [upload=12513]: Uploading segments... Aug 7 21:44:56.698908173 (12h5m31.214010758s) [upload=12513]: Got next segment. Inline: true Aug 7 21:44:56.698912338 (12h5m31.214014914s) [upload=12513]: Waiting for error group managing segments... Aug 7 21:44:56.698948274 (12h5m31.21405085s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:56.698965173 (12h5m31.214067748s) [upload=12513]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:56.698967987 (12h5m31.214070563s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:56.781273042 (12h5m31.296375616s) [upload=12513]: Done uploading segments... Aug 7 21:44:56.781296311 (12h5m31.296398893s) [upload=12513]: Upload finished. err: <nil> Aug 7 21:46:10.925701802 (12h6m45.44080437s) [upload=12532]: Starting upload Aug 7 21:46:10.925757954 (12h6m45.440860524s) [upload=12532]: Done starting upload Aug 7 21:46:10.925819534 (12h6m45.440922115s) [upload=12532]: Uploading segments... Aug 7 21:46:10.925869829 (12h6m45.44097242s) [upload=12532]: Got next segment. Inline: true Aug 7 21:46:10.925876475 (12h6m45.440979053s) [upload=12532]: Waiting for error group managing segments... Aug 7 21:46:10.925925157 (12h6m45.441027735s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:10.925930047 (12h6m45.441032624s) [upload=12532]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:10.925933179 (12h6m45.441035756s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:40.893959940 (12h7m15.409062507s) [upload=12532]: Done uploading segments... Aug 7 21:46:40.894015791 (12h7m15.409118368s) [upload=12532]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:17.091619874 (12h7m51.606722441s) [upload=12546]: Starting upload Aug 7 21:47:17.091714186 (12h7m51.606816768s) [upload=12546]: Done starting upload Aug 7 21:47:17.091785367 (12h7m51.606887938s) [upload=12546]: Uploading segments... Aug 7 21:47:17.091863768 (12h7m51.606966354s) [upload=12546]: Got next segment. Inline: true Aug 7 21:47:17.091870088 (12h7m51.606972665s) [upload=12546]: Waiting for error group managing segments... Aug 7 21:47:17.091926672 (12h7m51.607029237s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:17.091932153 (12h7m51.607034729s) [upload=12546]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:17.091935528 (12h7m51.607038105s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:47.062137982 (12h8m21.577240549s) [upload=12546]: Done uploading segments... Aug 7 21:47:47.062157139 (12h8m21.577259705s) [upload=12546]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:55.978154740 (12h8m30.493257315s) [upload=12560]: Starting upload Aug 7 21:47:55.978228943 (12h8m30.493331529s) [upload=12560]: Done starting upload Aug 7 21:47:55.978311008 (12h8m30.493413589s) [upload=12560]: Uploading segments... Aug 7 21:47:55.978374285 (12h8m30.493476868s) [upload=12560]: Got next segment. Inline: true Aug 7 21:47:55.978378891 (12h8m30.493481468s) [upload=12560]: Waiting for error group managing segments... Aug 7 21:47:55.978431682 (12h8m30.493534258s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:55.978436762 (12h8m30.493539338s) [upload=12560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:55.978439848 (12h8m30.493542423s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:25.945081513 (12h9m0.460184093s) [upload=12560]: Done uploading segments... Aug 7 21:48:25.945103520 (12h9m0.460206103s) [upload=12560]: Upload finished. err: metaclient: context deadline exceeded Duration: 253.331821ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXX.....X...............] duration: 757.591935ms FAIL (2025-08-07 21:59:57 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 68549s) [upload=12537]: Done starting upload Aug 7 21:46:22.615396117 (12h6m57.130498679s) [upload=12537]: Uploading segments... Aug 7 21:46:22.615846476 (12h6m57.13094906s) [upload=12537]: Got next segment. Inline: false Aug 7 21:46:22.615968068 (12h6m57.131070645s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:22.615975064 (12h6m57.131077646s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:28.156865165 (12h7m2.671967741s) [seg_pos={0 0}, upload=12537]: Begin upload segment... Aug 7 21:46:28.156917513 (12h7m2.672020096s) [seg_pos={0 0}, upload=12537]: Done begin upload segment. Aug 7 21:46:28.156967463 (12h7m2.672070048s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157000064 (12h7m2.672102649s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157067642 (12h7m2.672170211s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157406065 (12h7m2.672508645s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157549310 (12h7m2.672651889s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157739828 (12h7m2.672842407s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157989895 (12h7m2.673092473s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158099904 (12h7m2.673202487s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158381871 (12h7m2.673484448s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158490401 (12h7m2.673592979s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158794943 (12h7m2.67389751s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158855471 (12h7m2.673958052s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159214922 (12h7m2.674317506s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159509642 (12h7m2.674612208s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161192583 (12h7m2.676295161s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161765903 (12h7m2.676868473s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162066100 (12h7m2.677168688s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162703851 (12h7m2.67780643s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.268596153 (12h7m2.783698719s) [upload=12537]: Next returned nil segment Aug 7 21:46:28.268617977 (12h7m2.783720553s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:46:28.279079135 (12h7m2.794181715s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.290690859 (12h7m2.805793441s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.292185740 (12h7m2.807288316s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.294418015 (12h7m2.80952059s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.295076589 (12h7m2.810179159s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.297491713 (12h7m2.812594291s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.303643978 (12h7m2.818746547s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.304764566 (12h7m2.819867135s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.315655209 (12h7m2.830757788s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.323593569 (12h7m2.838696138s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.324982970 (12h7m2.840085548s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325108866 (12h7m2.840211445s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325989884 (12h7m2.841092452s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361102046 (12h7m2.876204623s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361164635 (12h7m2.876267211s) [seg_pos={0 0}, upload=12537]: Segment reached optimal threshold of 14 pieces. Aug 7 21:46:28.361368094 (12h7m2.876470663s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361578723 (12h7m2.876681288s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eeQieeLS35wwm6vk8rwcTDPPQgYD9vVhVw8WRt84bduXbaW2j): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361874485 (12h7m2.876977051s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o): protocol: read tcp 159.223.143.39:41094->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361908521 (12h7m2.877011091s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d): protocol: read tcp 159.223.143.39:38696->104.36.82.38:4693: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361930237 (12h7m2.877032803s) [seg_pos={0 0}, upload=12537]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:46:28.362049899 (12h7m2.877152488s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:28.362055767 (12h7m2.877158344s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:22.628005735 (12h7m57.143108314s) [upload=12537]: Done uploading segments... Aug 7 21:47:22.628037085 (12h7m57.143139659s) [upload=12537]: Upload finished. err: metaclient: context deadline exceeded Duration: 757.617923ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....XX................] duration: 420.722392ms FAIL (2025-08-07 21:59:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.800464ms edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [...............X.........XX.................................] duration: 33.794115ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:59:53 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 33.823152ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.X................] duration: 502.693254ms FAIL (2025-08-07 21:59:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 502.71309ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.018153603s FAIL (2025-08-07 21:59:51 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 1.018179083s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [............XXX........XX..XX...............................] duration: 150.5607ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:59:45 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 150.680376ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...X................] duration: 849.662131ms FAIL (2025-08-07 21:59:44 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 850.257314ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX............] duration: 501.883772ms FAIL (2025-08-07 21:59:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=12497]: Got next segment. Inline: false Aug 7 21:43:52.199899367 (12h4m26.715001952s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:52.199906793 (12h4m26.715009476s) [upload=12497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:22.166742883 (12h4m56.681845464s) [upload=12497]: Next segment err: context deadline exceeded Aug 7 21:44:22.166864816 (12h4m56.681967396s) [upload=12497]: Done uploading segments... Aug 7 21:44:22.166885734 (12h4m56.681988313s) [upload=12497]: Upload finished. err: context deadline exceeded Aug 7 21:45:03.330540016 (12h5m37.845642605s) [upload=12517]: Starting upload Aug 7 21:45:03.330650460 (12h5m37.845753069s) [upload=12517]: Done starting upload Aug 7 21:45:03.330858182 (12h5m37.845960762s) [upload=12517]: Uploading segments... Aug 7 21:45:03.331156974 (12h5m37.84625954s) [upload=12517]: Got next segment. Inline: false Aug 7 21:45:03.331235679 (12h5m37.846338242s) [upload=12517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:03.331241699 (12h5m37.846344282s) [upload=12517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:08.003810167 (12h5m42.518912754s) [seg_pos={0 0}, upload=12517]: Begin upload segment... Aug 7 21:45:08.003882230 (12h5m42.518984812s) [seg_pos={0 0}, upload=12517]: Done begin upload segment. Aug 7 21:45:08.003954509 (12h5m42.519057091s) [noise=true , address=77.90.31.13:4627 , piece_id=O4IROOSYED54KYRJPP6WSHWAGCMW64H2CDMOCH7EGDIBP7W7QVFQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.004263642 (12h5m42.51936623s) [noise=true , address=77.90.31.11:4555 , piece_id=Q3LP2RJ7MFDJABSDGNGWNYB27J2IORSCMA5BIFWZGAOSXF5A5BNQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.004792666 (12h5m42.519895256s) [noise=true , address=45.91.116.10:4083 , piece_id=LBBGLOISIFUGNPBVDLT4MZ5TFC4NTBDC6P2VT5IK6T75IIWEHWLA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.005354037 (12h5m42.520456625s) [noise=true , address=104.36.82.12:4149 , piece_id=57QOP6W7OHKTXBELK3LFIOFFMXD2TWMYSIWLI54URQRPNBLEJ5LA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.005841366 (12h5m42.520943955s) [noise=true , address=104.36.82.8:4969 , piece_id=BSBAVTYONQSVQBD2UQDDXZABBR7TG2TO6ADNCMUDM5A4IZV45UWA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.006328034 (12h5m42.521430625s) [noise=true , address=104.36.82.9:4007 , piece_id=WROTDT7DJG2HGV5UWALZRIZMSGMTH3N7SZ4F7SEI64GSIRLTU2LQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.006815458 (12h5m42.521918046s) [noise=true , address=104.36.82.6:4875 , piece_id=LPFAIF7C6W6YFDM5CHKFHSKWU6EMU4YWW7CSVZZRHD5GCM5BPALA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007331300 (12h5m42.522433909s) [noise=true , address=45.91.116.6:4975 , piece_id=5GDO73EDLMXIVE3OD7FDRAD3LEM5TCALGYSMB5QZNGYSN4I2SCSQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007697466 (12h5m42.522800074s) [noise=true , address=45.91.116.9:4263 , piece_id=64YKT5YSI7RURYDHZGB6SJIHJ6MBXWGE7UVIH6JGVA4XN4HT4XPQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007834257 (12h5m42.522936846s) [noise=true , address=77.90.31.14:4662 , piece_id=NOKVL5CJRIDTPCYOXQCITJO3KRXURIHXTUYEKSLEDWIOV3J6MSCQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.008267290 (12h5m42.523369892s) [noise=true , address=104.36.82.7:4912 , piece_id=TT3R2ZW7RBCVNZJ4YILZROCLUSR5IH6VUUQQCGR5G4O3GQC4JN6A, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.011188107 (12h5m42.52629075s) [noise=true , address=45.91.116.5:4297 , piece_id=MEGO3DATMLAO6UM3V7DPFXWPHVAQWS5I772LC2XQR5G7T6QRPWRA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.012274365 (12h5m42.527376954s) [noise=true , address=45.91.116.10:4082 , piece_id=Z6GB3XGMTCDAQMK2QZWJOGBLAUKLYVOLS6UGRWF4RLY37GVQVRBA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.012957723 (12h5m42.528060311s) [noise=true , address=104.36.82.4:4804 , piece_id=A5LKRZHVGBGBJTAJHJDDG54VOD6F5V2I7A3SWN24YR26JAPTX6WA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.014736176 (12h5m42.529838763s) [noise=true , address=77.90.31.11:4553 , piece_id=567MLMX4HQYW7QN5XNITKR47LJY56IPYJ6GLZYBYPILFYXLPR2JQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.016257825 (12h5m42.531360412s) [noise=true , address=104.36.82.10:4078 , piece_id=HI5SU6YHHN2OQF672MSMGY6RPETV5TCFHGJKI5AGN7NCZO47OXXQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.117745239 (12h5m42.632847808s) [upload=12517]: Next returned nil segment Aug 7 21:45:08.117761018 (12h5m42.632863594s) [upload=12517]: Waiting for error group managing segments... Aug 7 21:45:08.129666998 (12h5m42.644769574s) [noise=true , address=45.91.116.10:4082 , piece_id=Z6GB3XGMTCDAQMK2QZWJOGBLAUKLYVOLS6UGRWF4RLY37GVQVRBA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.130196788 (12h5m42.645299359s) [noise=true , address=77.90.31.11:4555 , piece_id=Q3LP2RJ7MFDJABSDGNGWNYB27J2IORSCMA5BIFWZGAOSXF5A5BNQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.130727325 (12h5m42.645829895s) [noise=true , address=77.90.31.11:4553 , piece_id=567MLMX4HQYW7QN5XNITKR47LJY56IPYJ6GLZYBYPILFYXLPR2JQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.132728107 (12h5m42.647830677s) [noise=true , address=77.90.31.14:4662 , piece_id=NOKVL5CJRIDTPCYOXQCITJO3KRXURIHXTUYEKSLEDWIOV3J6MSCQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.138647804 (12h5m42.653750375s) [noise=true , address=104.36.82.9:4007 , piece_id=WROTDT7DJG2HGV5UWALZRIZMSGMTH3N7SZ4F7SEI64GSIRLTU2LQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.141221377 (12h5m42.656323943s) [noise=true , address=104.36.82.12:4149 , piece_id=57QOP6W7OHKTXBELK3LFIOFFMXD2TWMYSIWLI54URQRPNBLEJ5LA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.142848205 (12h5m42.657950782s) [noise=true , address=45.91.116.5:4297 , piece_id=MEGO3DATMLAO6UM3V7DPFXWPHVAQWS5I772LC2XQR5G7T6QRPWRA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.145051855 (12h5m42.66015442s) [noise=true , address=45.91.116.9:4263 , piece_id=64YKT5YSI7RURYDHZGB6SJIHJ6MBXWGE7UVIH6JGVA4XN4HT4XPQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.145420726 (12h5m42.660523304s) [noise=true , address=45.91.116.6:4975 , piece_id=5GDO73EDLMXIVE3OD7FDRAD3LEM5TCALGYSMB5QZNGYSN4I2SCSQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.146452857 (12h5m42.661555425s) [noise=true , address=104.36.82.8:4969 , piece_id=BSBAVTYONQSVQBD2UQDDXZABBR7TG2TO6ADNCMUDM5A4IZV45UWA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.147080237 (12h5m42.6621828s) [noise=true , address=77.90.31.13:4627 , piece_id=O4IROOSYED54KYRJPP6WSHWAGCMW64H2CDMOCH7EGDIBP7W7QVFQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.148870189 (12h5m42.663972756s) [noise=true , address=104.36.82.4:4804 , piece_id=A5LKRZHVGBGBJTAJHJDDG54VOD6F5V2I7A3SWN24YR26JAPTX6WA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.151158066 (12h5m42.666260643s) [noise=true , address=45.91.116.10:4083 , piece_id=LBBGLOISIFUGNPBVDLT4MZ5TFC4NTBDC6P2VT5IK6T75IIWEHWLA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.152266060 (12h5m42.667368644s) [seg_pos={0 0}, upload=12517]: Segment reached optimal threshold of 13 pieces. Aug 7 21:45:08.152853862 (12h5m42.667956424s) [noise=true , address=104.36.82.7:4912 , piece_id=TT3R2ZW7RBCVNZJ4YILZROCLUSR5IH6VUUQQCGR5G4O3GQC4JN6A, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection; write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection Aug 7 21:45:08.153127403 (12h5m42.668229969s) [noise=true , address=104.36.82.6:4875 , piece_id=LPFAIF7C6W6YFDM5CHKFHSKWU6EMU4YWW7CSVZZRHD5GCM5BPALA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection; write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection Aug 7 21:45:08.154643012 (12h5m42.669745579s) [noise=true , address=104.36.82.10:4078 , piece_id=HI5SU6YHHN2OQF672MSMGY6RPETV5TCFHGJKI5AGN7NCZO47OXXQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 21:45:08.154667840 (12h5m42.669770416s) [seg_pos={0 0}, upload=12517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:45:08.154819614 (12h5m42.669922193s) [upload=12517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:45:08.154825838 (12h5m42.66992865s) [upload=12517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:00.396543819 (12h10m34.911646387s) [upload=12517]: Done uploading segments... Aug 7 21:50:00.396643233 (12h10m34.911745799s) [upload=12517]: Upload finished. err: metaclient: context deadline exceeded Duration: 501.90857ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...XXX.................] duration: 251.109574ms FAIL (2025-08-07 21:59:33 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:38.822741159 (12h4m13.337843727s) [upload=12496]: Starting upload Aug 7 21:43:38.822791612 (12h4m13.337894198s) [upload=12496]: Done starting upload Aug 7 21:43:38.822841316 (12h4m13.337943884s) [upload=12496]: Uploading segments... Aug 7 21:43:38.822878518 (12h4m13.337981103s) [upload=12496]: Got next segment. Inline: true Aug 7 21:43:38.822883293 (12h4m13.337985869s) [upload=12496]: Waiting for error group managing segments... Aug 7 21:43:38.822922170 (12h4m13.338024807s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:38.822926518 (12h4m13.338029093s) [upload=12496]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:38.822929823 (12h4m13.338032399s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:38.880695407 (12h4m13.395797984s) [upload=12496]: Done uploading segments... Aug 7 21:43:38.880716578 (12h4m13.395819144s) [upload=12496]: Upload finished. err: <nil> Aug 7 21:44:03.077928978 (12h4m37.593031579s) [upload=12501]: Starting upload Aug 7 21:44:03.078032002 (12h4m37.593134589s) [upload=12501]: Done starting upload Aug 7 21:44:03.078182274 (12h4m37.593284842s) [upload=12501]: Uploading segments... Aug 7 21:44:03.078254104 (12h4m37.59335669s) [upload=12501]: Got next segment. Inline: true Aug 7 21:44:03.078260761 (12h4m37.593363344s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:44:03.078324988 (12h4m37.593427564s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:03.078329732 (12h4m37.593432308s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:03.078333203 (12h4m37.593435779s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:03.150588344 (12h4m37.665690921s) [upload=12501]: Done uploading segments... Aug 7 21:44:03.150609592 (12h4m37.665712177s) [upload=12501]: Upload finished. err: <nil> Aug 7 21:44:56.698719656 (12h5m31.213822219s) [upload=12513]: Starting upload Aug 7 21:44:56.698786775 (12h5m31.213889351s) [upload=12513]: Done starting upload Aug 7 21:44:56.698856755 (12h5m31.213959317s) [upload=12513]: Uploading segments... Aug 7 21:44:56.698908173 (12h5m31.214010758s) [upload=12513]: Got next segment. Inline: true Aug 7 21:44:56.698912338 (12h5m31.214014914s) [upload=12513]: Waiting for error group managing segments... Aug 7 21:44:56.698948274 (12h5m31.21405085s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:56.698965173 (12h5m31.214067748s) [upload=12513]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:56.698967987 (12h5m31.214070563s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:56.781273042 (12h5m31.296375616s) [upload=12513]: Done uploading segments... Aug 7 21:44:56.781296311 (12h5m31.296398893s) [upload=12513]: Upload finished. err: <nil> Aug 7 21:46:10.925701802 (12h6m45.44080437s) [upload=12532]: Starting upload Aug 7 21:46:10.925757954 (12h6m45.440860524s) [upload=12532]: Done starting upload Aug 7 21:46:10.925819534 (12h6m45.440922115s) [upload=12532]: Uploading segments... Aug 7 21:46:10.925869829 (12h6m45.44097242s) [upload=12532]: Got next segment. Inline: true Aug 7 21:46:10.925876475 (12h6m45.440979053s) [upload=12532]: Waiting for error group managing segments... Aug 7 21:46:10.925925157 (12h6m45.441027735s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:10.925930047 (12h6m45.441032624s) [upload=12532]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:10.925933179 (12h6m45.441035756s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:40.893959940 (12h7m15.409062507s) [upload=12532]: Done uploading segments... Aug 7 21:46:40.894015791 (12h7m15.409118368s) [upload=12532]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:17.091619874 (12h7m51.606722441s) [upload=12546]: Starting upload Aug 7 21:47:17.091714186 (12h7m51.606816768s) [upload=12546]: Done starting upload Aug 7 21:47:17.091785367 (12h7m51.606887938s) [upload=12546]: Uploading segments... Aug 7 21:47:17.091863768 (12h7m51.606966354s) [upload=12546]: Got next segment. Inline: true Aug 7 21:47:17.091870088 (12h7m51.606972665s) [upload=12546]: Waiting for error group managing segments... Aug 7 21:47:17.091926672 (12h7m51.607029237s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:17.091932153 (12h7m51.607034729s) [upload=12546]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:17.091935528 (12h7m51.607038105s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:47.062137982 (12h8m21.577240549s) [upload=12546]: Done uploading segments... Aug 7 21:47:47.062157139 (12h8m21.577259705s) [upload=12546]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:55.978154740 (12h8m30.493257315s) [upload=12560]: Starting upload Aug 7 21:47:55.978228943 (12h8m30.493331529s) [upload=12560]: Done starting upload Aug 7 21:47:55.978311008 (12h8m30.493413589s) [upload=12560]: Uploading segments... Aug 7 21:47:55.978374285 (12h8m30.493476868s) [upload=12560]: Got next segment. Inline: true Aug 7 21:47:55.978378891 (12h8m30.493481468s) [upload=12560]: Waiting for error group managing segments... Aug 7 21:47:55.978431682 (12h8m30.493534258s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:55.978436762 (12h8m30.493539338s) [upload=12560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:55.978439848 (12h8m30.493542423s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:25.945081513 (12h9m0.460184093s) [upload=12560]: Done uploading segments... Aug 7 21:48:25.945103520 (12h9m0.460206103s) [upload=12560]: Upload finished. err: metaclient: context deadline exceeded Duration: 251.139067ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 470.951853ms FAIL (2025-08-07 21:59:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 470.9819ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.X...............] duration: 551.557803ms FAIL (2025-08-07 21:59:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 551.634736ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....XX...............] duration: 270.791658ms FAIL (2025-08-07 21:59:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 270.812914ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 30 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..................] duration: 216.694684ms FAIL (2025-08-07 21:59:21 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:02.683404470 (12h4m37.198507059s) [upload=12500]: Starting upload Aug 7 21:44:02.683498983 (12h4m37.198601564s) [upload=12500]: Done starting upload Aug 7 21:44:02.683587655 (12h4m37.198690226s) [upload=12500]: Uploading segments... Aug 7 21:44:02.683635288 (12h4m37.198737882s) [upload=12500]: Got next segment. Inline: true Aug 7 21:44:02.683641097 (12h4m37.198743679s) [upload=12500]: Waiting for error group managing segments... Aug 7 21:44:02.683702304 (12h4m37.198804891s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:02.683709172 (12h4m37.198811757s) [upload=12500]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:02.683713383 (12h4m37.198815961s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.749796670 (12h4m37.264899247s) [upload=12500]: Done uploading segments... Aug 7 21:44:02.749821074 (12h4m37.264923648s) [upload=12500]: Upload finished. err: <nil> Aug 7 21:44:59.056137042 (12h5m33.571239616s) [upload=12515]: Starting upload Aug 7 21:44:59.056204060 (12h5m33.571306636s) [upload=12515]: Done starting upload Aug 7 21:44:59.056275553 (12h5m33.571378122s) [upload=12515]: Uploading segments... Aug 7 21:44:59.056357038 (12h5m33.571459625s) [upload=12515]: Got next segment. Inline: true Aug 7 21:44:59.056364363 (12h5m33.57146694s) [upload=12515]: Waiting for error group managing segments... Aug 7 21:44:59.056439881 (12h5m33.571542451s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:59.056447251 (12h5m33.571549833s) [upload=12515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:59.056453265 (12h5m33.571555847s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:59.018996840 (12h6m33.534099417s) [upload=12515]: Done uploading segments... Aug 7 21:45:59.019015646 (12h6m33.534118222s) [upload=12515]: Upload finished. err: metaclient: context deadline exceeded Duration: 216.748025ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXX.....X..............] duration: 680.496484ms FAIL (2025-08-07 21:59:13 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 68549s) [upload=12537]: Done starting upload Aug 7 21:46:22.615396117 (12h6m57.130498679s) [upload=12537]: Uploading segments... Aug 7 21:46:22.615846476 (12h6m57.13094906s) [upload=12537]: Got next segment. Inline: false Aug 7 21:46:22.615968068 (12h6m57.131070645s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:22.615975064 (12h6m57.131077646s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:28.156865165 (12h7m2.671967741s) [seg_pos={0 0}, upload=12537]: Begin upload segment... Aug 7 21:46:28.156917513 (12h7m2.672020096s) [seg_pos={0 0}, upload=12537]: Done begin upload segment. Aug 7 21:46:28.156967463 (12h7m2.672070048s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157000064 (12h7m2.672102649s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157067642 (12h7m2.672170211s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157406065 (12h7m2.672508645s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157549310 (12h7m2.672651889s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157739828 (12h7m2.672842407s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157989895 (12h7m2.673092473s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158099904 (12h7m2.673202487s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158381871 (12h7m2.673484448s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158490401 (12h7m2.673592979s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158794943 (12h7m2.67389751s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158855471 (12h7m2.673958052s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159214922 (12h7m2.674317506s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159509642 (12h7m2.674612208s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161192583 (12h7m2.676295161s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161765903 (12h7m2.676868473s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162066100 (12h7m2.677168688s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162703851 (12h7m2.67780643s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.268596153 (12h7m2.783698719s) [upload=12537]: Next returned nil segment Aug 7 21:46:28.268617977 (12h7m2.783720553s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:46:28.279079135 (12h7m2.794181715s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.290690859 (12h7m2.805793441s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.292185740 (12h7m2.807288316s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.294418015 (12h7m2.80952059s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.295076589 (12h7m2.810179159s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.297491713 (12h7m2.812594291s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.303643978 (12h7m2.818746547s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.304764566 (12h7m2.819867135s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.315655209 (12h7m2.830757788s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.323593569 (12h7m2.838696138s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.324982970 (12h7m2.840085548s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325108866 (12h7m2.840211445s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325989884 (12h7m2.841092452s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361102046 (12h7m2.876204623s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361164635 (12h7m2.876267211s) [seg_pos={0 0}, upload=12537]: Segment reached optimal threshold of 14 pieces. Aug 7 21:46:28.361368094 (12h7m2.876470663s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361578723 (12h7m2.876681288s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eeQieeLS35wwm6vk8rwcTDPPQgYD9vVhVw8WRt84bduXbaW2j): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361874485 (12h7m2.876977051s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o): protocol: read tcp 159.223.143.39:41094->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361908521 (12h7m2.877011091s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d): protocol: read tcp 159.223.143.39:38696->104.36.82.38:4693: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361930237 (12h7m2.877032803s) [seg_pos={0 0}, upload=12537]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:46:28.362049899 (12h7m2.877152488s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:28.362055767 (12h7m2.877158344s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:22.628005735 (12h7m57.143108314s) [upload=12537]: Done uploading segments... Aug 7 21:47:22.628037085 (12h7m57.143139659s) [upload=12537]: Upload finished. err: metaclient: context deadline exceeded Duration: 680.527128ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.183371303s FAIL (2025-08-07 21:59:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 1.183398289s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 39 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...XXX................] duration: 493.462997ms FAIL (2025-08-07 21:59:10 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:38.822741159 (12h4m13.337843727s) [upload=12496]: Starting upload Aug 7 21:43:38.822791612 (12h4m13.337894198s) [upload=12496]: Done starting upload Aug 7 21:43:38.822841316 (12h4m13.337943884s) [upload=12496]: Uploading segments... Aug 7 21:43:38.822878518 (12h4m13.337981103s) [upload=12496]: Got next segment. Inline: true Aug 7 21:43:38.822883293 (12h4m13.337985869s) [upload=12496]: Waiting for error group managing segments... Aug 7 21:43:38.822922170 (12h4m13.338024807s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:38.822926518 (12h4m13.338029093s) [upload=12496]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:38.822929823 (12h4m13.338032399s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:38.880695407 (12h4m13.395797984s) [upload=12496]: Done uploading segments... Aug 7 21:43:38.880716578 (12h4m13.395819144s) [upload=12496]: Upload finished. err: <nil> Aug 7 21:44:03.077928978 (12h4m37.593031579s) [upload=12501]: Starting upload Aug 7 21:44:03.078032002 (12h4m37.593134589s) [upload=12501]: Done starting upload Aug 7 21:44:03.078182274 (12h4m37.593284842s) [upload=12501]: Uploading segments... Aug 7 21:44:03.078254104 (12h4m37.59335669s) [upload=12501]: Got next segment. Inline: true Aug 7 21:44:03.078260761 (12h4m37.593363344s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:44:03.078324988 (12h4m37.593427564s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:03.078329732 (12h4m37.593432308s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:03.078333203 (12h4m37.593435779s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:03.150588344 (12h4m37.665690921s) [upload=12501]: Done uploading segments... Aug 7 21:44:03.150609592 (12h4m37.665712177s) [upload=12501]: Upload finished. err: <nil> Aug 7 21:44:56.698719656 (12h5m31.213822219s) [upload=12513]: Starting upload Aug 7 21:44:56.698786775 (12h5m31.213889351s) [upload=12513]: Done starting upload Aug 7 21:44:56.698856755 (12h5m31.213959317s) [upload=12513]: Uploading segments... Aug 7 21:44:56.698908173 (12h5m31.214010758s) [upload=12513]: Got next segment. Inline: true Aug 7 21:44:56.698912338 (12h5m31.214014914s) [upload=12513]: Waiting for error group managing segments... Aug 7 21:44:56.698948274 (12h5m31.21405085s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:56.698965173 (12h5m31.214067748s) [upload=12513]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:56.698967987 (12h5m31.214070563s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:56.781273042 (12h5m31.296375616s) [upload=12513]: Done uploading segments... Aug 7 21:44:56.781296311 (12h5m31.296398893s) [upload=12513]: Upload finished. err: <nil> Aug 7 21:46:10.925701802 (12h6m45.44080437s) [upload=12532]: Starting upload Aug 7 21:46:10.925757954 (12h6m45.440860524s) [upload=12532]: Done starting upload Aug 7 21:46:10.925819534 (12h6m45.440922115s) [upload=12532]: Uploading segments... Aug 7 21:46:10.925869829 (12h6m45.44097242s) [upload=12532]: Got next segment. Inline: true Aug 7 21:46:10.925876475 (12h6m45.440979053s) [upload=12532]: Waiting for error group managing segments... Aug 7 21:46:10.925925157 (12h6m45.441027735s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:10.925930047 (12h6m45.441032624s) [upload=12532]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:10.925933179 (12h6m45.441035756s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:40.893959940 (12h7m15.409062507s) [upload=12532]: Done uploading segments... Aug 7 21:46:40.894015791 (12h7m15.409118368s) [upload=12532]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:17.091619874 (12h7m51.606722441s) [upload=12546]: Starting upload Aug 7 21:47:17.091714186 (12h7m51.606816768s) [upload=12546]: Done starting upload Aug 7 21:47:17.091785367 (12h7m51.606887938s) [upload=12546]: Uploading segments... Aug 7 21:47:17.091863768 (12h7m51.606966354s) [upload=12546]: Got next segment. Inline: true Aug 7 21:47:17.091870088 (12h7m51.606972665s) [upload=12546]: Waiting for error group managing segments... Aug 7 21:47:17.091926672 (12h7m51.607029237s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:17.091932153 (12h7m51.607034729s) [upload=12546]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:17.091935528 (12h7m51.607038105s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:47.062137982 (12h8m21.577240549s) [upload=12546]: Done uploading segments... Aug 7 21:47:47.062157139 (12h8m21.577259705s) [upload=12546]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:55.978154740 (12h8m30.493257315s) [upload=12560]: Starting upload Aug 7 21:47:55.978228943 (12h8m30.493331529s) [upload=12560]: Done starting upload Aug 7 21:47:55.978311008 (12h8m30.493413589s) [upload=12560]: Uploading segments... Aug 7 21:47:55.978374285 (12h8m30.493476868s) [upload=12560]: Got next segment. Inline: true Aug 7 21:47:55.978378891 (12h8m30.493481468s) [upload=12560]: Waiting for error group managing segments... Aug 7 21:47:55.978431682 (12h8m30.493534258s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:55.978436762 (12h8m30.493539338s) [upload=12560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:55.978439848 (12h8m30.493542423s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:25.945081513 (12h9m0.460184093s) [upload=12560]: Done uploading segments... Aug 7 21:48:25.945103520 (12h9m0.460206103s) [upload=12560]: Upload finished. err: metaclient: context deadline exceeded Duration: 493.486971ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXX.XX.X..X................] duration: 2.007144889s FAIL (2025-08-07 21:59:10 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.007171409s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [........X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX...........] duration: 627.67472ms FAIL (2025-08-07 21:59:01 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=12497]: Got next segment. Inline: false Aug 7 21:43:52.199899367 (12h4m26.715001952s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:52.199906793 (12h4m26.715009476s) [upload=12497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:22.166742883 (12h4m56.681845464s) [upload=12497]: Next segment err: context deadline exceeded Aug 7 21:44:22.166864816 (12h4m56.681967396s) [upload=12497]: Done uploading segments... Aug 7 21:44:22.166885734 (12h4m56.681988313s) [upload=12497]: Upload finished. err: context deadline exceeded Aug 7 21:45:03.330540016 (12h5m37.845642605s) [upload=12517]: Starting upload Aug 7 21:45:03.330650460 (12h5m37.845753069s) [upload=12517]: Done starting upload Aug 7 21:45:03.330858182 (12h5m37.845960762s) [upload=12517]: Uploading segments... Aug 7 21:45:03.331156974 (12h5m37.84625954s) [upload=12517]: Got next segment. Inline: false Aug 7 21:45:03.331235679 (12h5m37.846338242s) [upload=12517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:03.331241699 (12h5m37.846344282s) [upload=12517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:08.003810167 (12h5m42.518912754s) [seg_pos={0 0}, upload=12517]: Begin upload segment... Aug 7 21:45:08.003882230 (12h5m42.518984812s) [seg_pos={0 0}, upload=12517]: Done begin upload segment. Aug 7 21:45:08.003954509 (12h5m42.519057091s) [noise=true , address=77.90.31.13:4627 , piece_id=O4IROOSYED54KYRJPP6WSHWAGCMW64H2CDMOCH7EGDIBP7W7QVFQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.004263642 (12h5m42.51936623s) [noise=true , address=77.90.31.11:4555 , piece_id=Q3LP2RJ7MFDJABSDGNGWNYB27J2IORSCMA5BIFWZGAOSXF5A5BNQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.004792666 (12h5m42.519895256s) [noise=true , address=45.91.116.10:4083 , piece_id=LBBGLOISIFUGNPBVDLT4MZ5TFC4NTBDC6P2VT5IK6T75IIWEHWLA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.005354037 (12h5m42.520456625s) [noise=true , address=104.36.82.12:4149 , piece_id=57QOP6W7OHKTXBELK3LFIOFFMXD2TWMYSIWLI54URQRPNBLEJ5LA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.005841366 (12h5m42.520943955s) [noise=true , address=104.36.82.8:4969 , piece_id=BSBAVTYONQSVQBD2UQDDXZABBR7TG2TO6ADNCMUDM5A4IZV45UWA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.006328034 (12h5m42.521430625s) [noise=true , address=104.36.82.9:4007 , piece_id=WROTDT7DJG2HGV5UWALZRIZMSGMTH3N7SZ4F7SEI64GSIRLTU2LQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.006815458 (12h5m42.521918046s) [noise=true , address=104.36.82.6:4875 , piece_id=LPFAIF7C6W6YFDM5CHKFHSKWU6EMU4YWW7CSVZZRHD5GCM5BPALA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007331300 (12h5m42.522433909s) [noise=true , address=45.91.116.6:4975 , piece_id=5GDO73EDLMXIVE3OD7FDRAD3LEM5TCALGYSMB5QZNGYSN4I2SCSQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007697466 (12h5m42.522800074s) [noise=true , address=45.91.116.9:4263 , piece_id=64YKT5YSI7RURYDHZGB6SJIHJ6MBXWGE7UVIH6JGVA4XN4HT4XPQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.007834257 (12h5m42.522936846s) [noise=true , address=77.90.31.14:4662 , piece_id=NOKVL5CJRIDTPCYOXQCITJO3KRXURIHXTUYEKSLEDWIOV3J6MSCQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.008267290 (12h5m42.523369892s) [noise=true , address=104.36.82.7:4912 , piece_id=TT3R2ZW7RBCVNZJ4YILZROCLUSR5IH6VUUQQCGR5G4O3GQC4JN6A, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.011188107 (12h5m42.52629075s) [noise=true , address=45.91.116.5:4297 , piece_id=MEGO3DATMLAO6UM3V7DPFXWPHVAQWS5I772LC2XQR5G7T6QRPWRA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.012274365 (12h5m42.527376954s) [noise=true , address=45.91.116.10:4082 , piece_id=Z6GB3XGMTCDAQMK2QZWJOGBLAUKLYVOLS6UGRWF4RLY37GVQVRBA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.012957723 (12h5m42.528060311s) [noise=true , address=104.36.82.4:4804 , piece_id=A5LKRZHVGBGBJTAJHJDDG54VOD6F5V2I7A3SWN24YR26JAPTX6WA, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.014736176 (12h5m42.529838763s) [noise=true , address=77.90.31.11:4553 , piece_id=567MLMX4HQYW7QN5XNITKR47LJY56IPYJ6GLZYBYPILFYXLPR2JQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.016257825 (12h5m42.531360412s) [noise=true , address=104.36.82.10:4078 , piece_id=HI5SU6YHHN2OQF672MSMGY6RPETV5TCFHGJKI5AGN7NCZO47OXXQ, seg_pos={0 0}, upload=12517]: Uploading piece... Aug 7 21:45:08.117745239 (12h5m42.632847808s) [upload=12517]: Next returned nil segment Aug 7 21:45:08.117761018 (12h5m42.632863594s) [upload=12517]: Waiting for error group managing segments... Aug 7 21:45:08.129666998 (12h5m42.644769574s) [noise=true , address=45.91.116.10:4082 , piece_id=Z6GB3XGMTCDAQMK2QZWJOGBLAUKLYVOLS6UGRWF4RLY37GVQVRBA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.130196788 (12h5m42.645299359s) [noise=true , address=77.90.31.11:4555 , piece_id=Q3LP2RJ7MFDJABSDGNGWNYB27J2IORSCMA5BIFWZGAOSXF5A5BNQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.130727325 (12h5m42.645829895s) [noise=true , address=77.90.31.11:4553 , piece_id=567MLMX4HQYW7QN5XNITKR47LJY56IPYJ6GLZYBYPILFYXLPR2JQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.132728107 (12h5m42.647830677s) [noise=true , address=77.90.31.14:4662 , piece_id=NOKVL5CJRIDTPCYOXQCITJO3KRXURIHXTUYEKSLEDWIOV3J6MSCQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.138647804 (12h5m42.653750375s) [noise=true , address=104.36.82.9:4007 , piece_id=WROTDT7DJG2HGV5UWALZRIZMSGMTH3N7SZ4F7SEI64GSIRLTU2LQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.141221377 (12h5m42.656323943s) [noise=true , address=104.36.82.12:4149 , piece_id=57QOP6W7OHKTXBELK3LFIOFFMXD2TWMYSIWLI54URQRPNBLEJ5LA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.142848205 (12h5m42.657950782s) [noise=true , address=45.91.116.5:4297 , piece_id=MEGO3DATMLAO6UM3V7DPFXWPHVAQWS5I772LC2XQR5G7T6QRPWRA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.145051855 (12h5m42.66015442s) [noise=true , address=45.91.116.9:4263 , piece_id=64YKT5YSI7RURYDHZGB6SJIHJ6MBXWGE7UVIH6JGVA4XN4HT4XPQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.145420726 (12h5m42.660523304s) [noise=true , address=45.91.116.6:4975 , piece_id=5GDO73EDLMXIVE3OD7FDRAD3LEM5TCALGYSMB5QZNGYSN4I2SCSQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.146452857 (12h5m42.661555425s) [noise=true , address=104.36.82.8:4969 , piece_id=BSBAVTYONQSVQBD2UQDDXZABBR7TG2TO6ADNCMUDM5A4IZV45UWA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.147080237 (12h5m42.6621828s) [noise=true , address=77.90.31.13:4627 , piece_id=O4IROOSYED54KYRJPP6WSHWAGCMW64H2CDMOCH7EGDIBP7W7QVFQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.148870189 (12h5m42.663972756s) [noise=true , address=104.36.82.4:4804 , piece_id=A5LKRZHVGBGBJTAJHJDDG54VOD6F5V2I7A3SWN24YR26JAPTX6WA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.151158066 (12h5m42.666260643s) [noise=true , address=45.91.116.10:4083 , piece_id=LBBGLOISIFUGNPBVDLT4MZ5TFC4NTBDC6P2VT5IK6T75IIWEHWLA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: <nil> Aug 7 21:45:08.152266060 (12h5m42.667368644s) [seg_pos={0 0}, upload=12517]: Segment reached optimal threshold of 13 pieces. Aug 7 21:45:08.152853862 (12h5m42.667956424s) [noise=true , address=104.36.82.7:4912 , piece_id=TT3R2ZW7RBCVNZJ4YILZROCLUSR5IH6VUUQQCGR5G4O3GQC4JN6A, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection; write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57550->104.36.82.7:4912: use of closed network connection Aug 7 21:45:08.153127403 (12h5m42.668229969s) [noise=true , address=104.36.82.6:4875 , piece_id=LPFAIF7C6W6YFDM5CHKFHSKWU6EMU4YWW7CSVZZRHD5GCM5BPALA, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection; write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45924->104.36.82.6:4875: use of closed network connection Aug 7 21:45:08.154643012 (12h5m42.669745579s) [noise=true , address=104.36.82.10:4078 , piece_id=HI5SU6YHHN2OQF672MSMGY6RPETV5TCFHGJKI5AGN7NCZO47OXXQ, seg_pos={0 0}, upload=12517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Aug 7 21:45:08.154667840 (12h5m42.669770416s) [seg_pos={0 0}, upload=12517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:45:08.154819614 (12h5m42.669922193s) [upload=12517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:45:08.154825838 (12h5m42.66992865s) [upload=12517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:00.396543819 (12h10m34.911646387s) [upload=12517]: Done uploading segments... Aug 7 21:50:00.396643233 (12h10m34.911745799s) [upload=12517]: Upload finished. err: metaclient: context deadline exceeded Duration: 627.704845ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..X..X.............] duration: 252.648503ms FAIL (2025-08-07 21:59:01 UTC) operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 252.671671ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...X...............] duration: 675.356524ms FAIL (2025-08-07 21:59:00 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 675.383519ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 30 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.................] duration: 287.419429ms FAIL (2025-08-07 21:58:49 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:02.683404470 (12h4m37.198507059s) [upload=12500]: Starting upload Aug 7 21:44:02.683498983 (12h4m37.198601564s) [upload=12500]: Done starting upload Aug 7 21:44:02.683587655 (12h4m37.198690226s) [upload=12500]: Uploading segments... Aug 7 21:44:02.683635288 (12h4m37.198737882s) [upload=12500]: Got next segment. Inline: true Aug 7 21:44:02.683641097 (12h4m37.198743679s) [upload=12500]: Waiting for error group managing segments... Aug 7 21:44:02.683702304 (12h4m37.198804891s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:02.683709172 (12h4m37.198811757s) [upload=12500]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:02.683713383 (12h4m37.198815961s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.749796670 (12h4m37.264899247s) [upload=12500]: Done uploading segments... Aug 7 21:44:02.749821074 (12h4m37.264923648s) [upload=12500]: Upload finished. err: <nil> Aug 7 21:44:59.056137042 (12h5m33.571239616s) [upload=12515]: Starting upload Aug 7 21:44:59.056204060 (12h5m33.571306636s) [upload=12515]: Done starting upload Aug 7 21:44:59.056275553 (12h5m33.571378122s) [upload=12515]: Uploading segments... Aug 7 21:44:59.056357038 (12h5m33.571459625s) [upload=12515]: Got next segment. Inline: true Aug 7 21:44:59.056364363 (12h5m33.57146694s) [upload=12515]: Waiting for error group managing segments... Aug 7 21:44:59.056439881 (12h5m33.571542451s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:59.056447251 (12h5m33.571549833s) [upload=12515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:59.056453265 (12h5m33.571555847s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:59.018996840 (12h6m33.534099417s) [upload=12515]: Done uploading segments... Aug 7 21:45:59.019015646 (12h6m33.534118222s) [upload=12515]: Upload finished. err: metaclient: context deadline exceeded Duration: 287.464116ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXX.XX.X..X...............] duration: 1.996233128s FAIL (2025-08-07 21:58:46 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.996263306s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.............XXX........XX..XX..............................] duration: 97.276723ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:58:46 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 97.300209ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [................X.........XX................................] duration: 47.249926ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:58:42 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 47.285195ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.X..............] duration: 598.570898ms FAIL (2025-08-07 21:58:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 598.591823ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXX.XX.X..X..............] duration: 1.716386746s FAIL (2025-08-07 21:58:19 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.71640673s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 409.935132ms FAIL (2025-08-07 21:58:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 409.960688ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..X..X............] duration: 344.64505ms FAIL (2025-08-07 21:58:15 UTC) operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 344.669252ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....XX..............] duration: 312.294066ms FAIL (2025-08-07 21:58:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 312.316095ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.X.............] duration: 648.689966ms FAIL (2025-08-07 21:58:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 648.719323ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)