Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 30.001138529s FAIL (2025-08-07 21:22:19 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001162562s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 40.78512ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:22:17 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 40.804271ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 10 out of 60 is failed (threshold=3) [.................................................X.XXXXXXXXX] duration: 30.001384781s FAIL (2025-08-07 21:22:00 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load Aug 7 21:13:34.490712022 (11h34m9.005814599s) [upload=12029]: Uploading segments... Aug 7 21:13:34.490822609 (11h34m9.005925188s) [upload=12029]: Got next segment. Inline: true Aug 7 21:13:34.490828860 (11h34m9.005931437s) [upload=12029]: Waiting for error group managing segments... Aug 7 21:13:34.490903638 (11h34m9.006006215s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:34.490908813 (11h34m9.006011379s) [upload=12029]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:34.490911892 (11h34m9.00601446s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:34.880488092 (11h34m9.39559068s) [upload=12029]: Done uploading segments... Aug 7 21:13:34.880508926 (11h34m9.395611515s) [upload=12029]: Upload finished. err: <nil> Aug 7 21:14:45.394850433 (11h35m19.909953011s) [upload=12049]: Starting upload Aug 7 21:14:45.394946608 (11h35m19.910049187s) [upload=12049]: Done starting upload Aug 7 21:14:45.395065277 (11h35m19.910167846s) [upload=12049]: Uploading segments... Aug 7 21:14:45.395104752 (11h35m19.910207337s) [upload=12049]: Got next segment. Inline: true Aug 7 21:14:45.395110169 (11h35m19.910212745s) [upload=12049]: Waiting for error group managing segments... Aug 7 21:14:45.395152641 (11h35m19.910255207s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.395157021 (11h35m19.910259598s) [upload=12049]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.395161401 (11h35m19.910263982s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.359522906 (11h35m49.874625488s) [upload=12049]: Done uploading segments... Aug 7 21:15:15.359549039 (11h35m49.874651622s) [upload=12049]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:53.553631580 (11h36m28.068734159s) [upload=12066]: Starting upload Aug 7 21:15:53.553693025 (11h36m28.068795593s) [upload=12066]: Done starting upload Aug 7 21:15:53.553757062 (11h36m28.06885963s) [upload=12066]: Uploading segments... Aug 7 21:15:53.553809335 (11h36m28.068911922s) [upload=12066]: Got next segment. Inline: true Aug 7 21:15:53.553813895 (11h36m28.068916471s) [upload=12066]: Waiting for error group managing segments... Aug 7 21:15:53.553863687 (11h36m28.068966263s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:53.553867873 (11h36m28.068970453s) [upload=12066]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:53.553871084 (11h36m28.068973661s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:23.520202496 (11h36m58.035305071s) [upload=12066]: Done uploading segments... Aug 7 21:16:23.520223689 (11h36m58.035326265s) [upload=12066]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:49.902511179 (11h37m24.417613746s) [upload=12081]: Starting upload Aug 7 21:16:49.902589681 (11h37m24.417692258s) [upload=12081]: Done starting upload Aug 7 21:16:49.902694562 (11h37m24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001407468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................................................X..XXXXXXX] duration: 30.000750412s FAIL (2025-08-07 21:21:53 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040942288 (11h35m39.556044876s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.041546451 (11h35m39.556649039s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042086243 (11h35m39.557188834s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042565982 (11h35m39.557668574s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043348608 (11h35m39.558451196s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043913089 (11h35m39.559015677s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.051646181 (11h35m39.566748771s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.177029777 (11h35m39.692132351s) [upload=12056]: Next returned nil segment Aug 7 21:15:05.177048224 (11h35m39.692150808s) [upload=12056]: Waiting for error group managing segments... Aug 7 21:15:05.181493366 (11h35m39.696595929s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182082323 (11h35m39.697184889s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182627665 (11h35m39.697730245s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183210775 (11h35m39.698313344s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183776547 (11h35m39.69887913s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184100393 (11h35m39.699202963s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184388388 (11h35m39.699490959s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185120907 (11h35m39.700223474s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185748114 (11h35m39.700850683s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces. Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments... Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments... Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments... Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments... Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments... Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments... Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments... Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments... Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments... Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded Duration: 30.000773929s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 1m0.001942098s FAIL (2025-08-07 21:21:53 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001977476s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 49.804797ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:21:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 49.825395ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 11 out of 60 is failed (threshold=3) [...............................................X..XXXXXXXXXX] duration: 30.000869418s FAIL (2025-08-07 21:21:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000891337s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 30.00044764s FAIL (2025-08-07 21:21:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000481987s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 1m0.000639149s FAIL (2025-08-07 21:21:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ug 7 21:11:30.967969502 (11h32m5.483072083s) [upload=11995]: Done starting upload Aug 7 21:11:30.968040172 (11h32m5.483142743s) [upload=11995]: Uploading segments... Aug 7 21:11:30.968088031 (11h32m5.483190618s) [upload=11995]: Got next segment. Inline: true Aug 7 21:11:30.968092912 (11h32m5.483195492s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:11:30.968156155 (11h32m5.483258717s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:30.968160963 (11h32m5.483263526s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:30.968163832 (11h32m5.483266409s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.040679973 (11h32m5.555782542s) [upload=11995]: Done uploading segments... Aug 7 21:11:31.040697828 (11h32m5.555800393s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:12:44.577632884 (11h33m19.092735462s) [upload=12014]: Starting upload Aug 7 21:12:44.577718424 (11h33m19.092821007s) [upload=12014]: Done starting upload Aug 7 21:12:44.577778064 (11h33m19.09288064s) [upload=12014]: Uploading segments... Aug 7 21:12:44.577870249 (11h33m19.09297311s) [upload=12014]: Got next segment. Inline: true Aug 7 21:12:44.577878347 (11h33m19.09298093s) [upload=12014]: Waiting for error group managing segments... Aug 7 21:12:44.577949813 (11h33m19.093052392s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:44.577956047 (11h33m19.093058626s) [upload=12014]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:44.577963216 (11h33m19.093065803s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:44.741155623 (11h33m19.256258199s) [upload=12014]: Done uploading segments... Aug 7 21:12:44.741173325 (11h33m19.256275891s) [upload=12014]: Upload finished. err: <nil> Aug 7 21:13:22.079763163 (11h33m56.594865742s) [upload=12027]: Starting upload Aug 7 21:13:22.079827372 (11h33m56.594929946s) [upload=12027]: Done starting upload Aug 7 21:13:22.079907596 (11h33m56.595010178s) [upload=12027]: Uploading segments... Aug 7 21:13:22.079966866 (11h33m56.595069445s) [upload=12027]: Got next segment. Inline: true Aug 7 21:13:22.079973866 (11h33m56.595076443s) [upload=12027]: Waiting for error group managing segments... Aug 7 21:13:22.080019901 (11h33m56.595122472s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:22.080024370 (11h33m56.595126947s) [upload=12027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:22.080029157 (11h33m56.595131732s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:22.199851686 (11h33m56.714954269s) [upload=12027]: Done uploading segments... Aug 7 21:13:22.199871202 (11h33m56.714973784s) [upload=12027]: Upload finished. err: <nil> Aug 7 21:14:11.619394074 (11h34m46.134496661s) [upload=12040]: Starting upload Aug 7 21:14:11.619503144 (11h34m46.134605739s) [upload=12040]: Done starting upload Aug 7 21:14:11.619622676 (11h34m46.13472524s) [upload=12040]: Uploading segments... Aug 7 21:14:11.619678842 (11h34m46.134781443s) [upload=12040]: Got next segment. Inline: true Aug 7 21:14:11.619698988 (11h34m46.134801579s) [upload=12040]: Waiting for error group managing segments... Aug 7 21:14:11.619745714 (11h34m46.1348483s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:11.619750849 (11h34m46.134853438s) [upload=12040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:11.619754200 (11h34m46.134856779s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:11.587599715 (11h35m46.102702292s) [upload=12040]: Done uploading segments... Aug 7 21:15:11.587622036 (11h35m46.102724617s) [upload=12040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:11.944029087 (11h35m46.459131673s) [upload=12059]: Starting upload Aug 7 21:15:11.944115611 (11h35m46.459218173s) [upload=12059]: Done starting upload Aug 7 21:15:11.944188828 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000658585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 30.001269502s FAIL (2025-08-07 21:21:25 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001294101s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [..................................................X.XXXXXXXX] duration: 30.0016516s FAIL (2025-08-07 21:21:12 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Aug 7 21:13:00.260688806 (11h33m34.775791377s) [upload=12021]: Uploading segments... Aug 7 21:13:00.260755956 (11h33m34.775858537s) [upload=12021]: Got next segment. Inline: true Aug 7 21:13:00.260760943 (11h33m34.775863519s) [upload=12021]: Waiting for error group managing segments... Aug 7 21:13:00.260825909 (11h33m34.775928474s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:00.260831124 (11h33m34.775933703s) [upload=12021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:00.260834322 (11h33m34.7759369s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:30.231998317 (11h34m4.747100894s) [upload=12021]: Done uploading segments... Aug 7 21:13:30.232017085 (11h34m4.74711966s) [upload=12021]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:34.490529998 (11h34m9.005632564s) [upload=12029]: Starting upload Aug 7 21:13:34.490599665 (11h34m9.005702244s) [upload=12029]: Done starting upload Aug 7 21:13:34.490712022 (11h34m9.005814599s) [upload=12029]: Uploading segments... Aug 7 21:13:34.490822609 (11h34m9.005925188s) [upload=12029]: Got next segment. Inline: true Aug 7 21:13:34.490828860 (11h34m9.005931437s) [upload=12029]: Waiting for error group managing segments... Aug 7 21:13:34.490903638 (11h34m9.006006215s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:34.490908813 (11h34m9.006011379s) [upload=12029]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:34.490911892 (11h34m9.00601446s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:34.880488092 (11h34m9.39559068s) [upload=12029]: Done uploading segments... Aug 7 21:13:34.880508926 (11h34m9.395611515s) [upload=12029]: Upload finished. err: <nil> Aug 7 21:14:45.394850433 (11h35m19.909953011s) [upload=12049]: Starting upload Aug 7 21:14:45.394946608 (11h35m19.910049187s) [upload=12049]: Done starting upload Aug 7 21:14:45.395065277 (11h35m19.910167846s) [upload=12049]: Uploading segments... Aug 7 21:14:45.395104752 (11h35m19.910207337s) [upload=12049]: Got next segment. Inline: true Aug 7 21:14:45.395110169 (11h35m19.910212745s) [upload=12049]: Waiting for error group managing segments... Aug 7 21:14:45.395152641 (11h35m19.910255207s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.395157021 (11h35m19.910259598s) [upload=12049]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.395161401 (11h35m19.910263982s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.359522906 (11h35m49.874625488s) [upload=12049]: Done uploading segments... Aug 7 21:15:15.359549039 (11h35m49.874651622s) [upload=12049]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:53.553631580 (11h36m28.068734159s) [upload=12066]: Starting upload Aug 7 21:15:53.553693025 (11h36m28.068795593s) [upload=12066]: Done starting upload Aug 7 21:15:53.553757062 (11h36m28.06885963s) [upload=12066]: Uploading segments... Aug 7 21:15:53.553809335 (11h36m28.068911922s) [upload=12066]: Got next segment. Inline: true Aug 7 21:15:53.553813895 (11h36m28.068916471s) [upload=12066]: Waiting for error group managing segments... Aug 7 21:15:53.553863687 (11h36m28.068966263s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:53.553867873 (11h36m28.068970453s) [upload=12066]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:53.553871084 (11h36m28.068973661s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:23.520202496 (11h36m58.035305071s) [upload=12066]: Done uploading segments... Aug 7 21:16:23.520223689 (11h36m58.035326265s) [upload=12066]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:49.902511179 (11h37m24.417613746s) [upload=12081]: Starting upload Aug 7 21:16:49.902589681 (11h37m24.417692258s) [upload=12081]: Done starting upload Aug 7 21:16:49.902694562 (11h37m24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001678354s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 44.591252ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:21:12 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 44.615812ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [................................................X..XXXXXXXXX] duration: 30.001277128s FAIL (2025-08-07 21:21:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001299575s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 30.001144344s FAIL (2025-08-07 21:21:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001165169s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................................................X..XXXXXX] duration: 30.00095221s FAIL (2025-08-07 21:21:00 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 86s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038728250 (11h35m39.553830831s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.039092766 (11h35m39.554195355s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.039689079 (11h35m39.554791667s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040315765 (11h35m39.555418329s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040942288 (11h35m39.556044876s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.041546451 (11h35m39.556649039s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042086243 (11h35m39.557188834s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042565982 (11h35m39.557668574s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043348608 (11h35m39.558451196s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043913089 (11h35m39.559015677s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.051646181 (11h35m39.566748771s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.177029777 (11h35m39.692132351s) [upload=12056]: Next returned nil segment Aug 7 21:15:05.177048224 (11h35m39.692150808s) [upload=12056]: Waiting for error group managing segments... Aug 7 21:15:05.181493366 (11h35m39.696595929s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182082323 (11h35m39.697184889s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182627665 (11h35m39.697730245s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183210775 (11h35m39.698313344s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183776547 (11h35m39.69887913s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184100393 (11h35m39.699202963s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184388388 (11h35m39.699490959s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185120907 (11h35m39.700223474s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185748114 (11h35m39.700850683s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces. Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments... Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments... Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments... Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments... Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments... Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments... Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments... Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded Duration: 30.000980702s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 1m0.000610649s FAIL (2025-08-07 21:20:53 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00062431s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 43.846119ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:20:44 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 43.863029ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.000433175s FAIL (2025-08-07 21:20:36 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=11989]: Starting upload Aug 7 21:11:07.821828021 (11h31m42.336930607s) [upload=11989]: Done starting upload Aug 7 21:11:07.821974268 (11h31m42.337076852s) [upload=11989]: Uploading segments... Aug 7 21:11:07.822051165 (11h31m42.33715375s) [upload=11989]: Got next segment. Inline: true Aug 7 21:11:07.822058710 (11h31m42.337161296s) [upload=11989]: Waiting for error group managing segments... Aug 7 21:11:07.822137123 (11h31m42.337239714s) [upload=11989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:07.822143163 (11h31m42.33724575s) [upload=11989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:07.822147451 (11h31m42.337250044s) [upload=11989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:07.880868203 (11h31m42.395970789s) [upload=11989]: Done uploading segments... Aug 7 21:11:07.880886229 (11h31m42.395988796s) [upload=11989]: Upload finished. err: <nil> Aug 7 21:11:30.967897030 (11h32m5.482999597s) [upload=11995]: Starting upload Aug 7 21:11:30.967969502 (11h32m5.483072083s) [upload=11995]: Done starting upload Aug 7 21:11:30.968040172 (11h32m5.483142743s) [upload=11995]: Uploading segments... Aug 7 21:11:30.968088031 (11h32m5.483190618s) [upload=11995]: Got next segment. Inline: true Aug 7 21:11:30.968092912 (11h32m5.483195492s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:11:30.968156155 (11h32m5.483258717s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:30.968160963 (11h32m5.483263526s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:30.968163832 (11h32m5.483266409s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.040679973 (11h32m5.555782542s) [upload=11995]: Done uploading segments... Aug 7 21:11:31.040697828 (11h32m5.555800393s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:12:44.577632884 (11h33m19.092735462s) [upload=12014]: Starting upload Aug 7 21:12:44.577718424 (11h33m19.092821007s) [upload=12014]: Done starting upload Aug 7 21:12:44.577778064 (11h33m19.09288064s) [upload=12014]: Uploading segments... Aug 7 21:12:44.577870249 (11h33m19.09297311s) [upload=12014]: Got next segment. Inline: true Aug 7 21:12:44.577878347 (11h33m19.09298093s) [upload=12014]: Waiting for error group managing segments... Aug 7 21:12:44.577949813 (11h33m19.093052392s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:44.577956047 (11h33m19.093058626s) [upload=12014]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:44.577963216 (11h33m19.093065803s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:44.741155623 (11h33m19.256258199s) [upload=12014]: Done uploading segments... Aug 7 21:12:44.741173325 (11h33m19.256275891s) [upload=12014]: Upload finished. err: <nil> Aug 7 21:13:22.079763163 (11h33m56.594865742s) [upload=12027]: Starting upload Aug 7 21:13:22.079827372 (11h33m56.594929946s) [upload=12027]: Done starting upload Aug 7 21:13:22.079907596 (11h33m56.595010178s) [upload=12027]: Uploading segments... Aug 7 21:13:22.079966866 (11h33m56.595069445s) [upload=12027]: Got next segment. Inline: true Aug 7 21:13:22.079973866 (11h33m56.595076443s) [upload=12027]: Waiting for error group managing segments... Aug 7 21:13:22.080019901 (11h33m56.595122472s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:22.080024370 (11h33m56.595126947s) [upload=12027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:22.080029157 (11h33m56.595131732s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:22.199851686 (11h33m56.714954269s) [upload=12027]: Done uploading segments... Aug 7 21:13:22.199871202 (11h33m56.714973784s) [upload=12027]: Upload finished. err: <nil> Aug 7 21:14:11.619394074 (11h34m46.134496661s) [upload=12040]: Starting upload Aug 7 21:14:11.619503144 (11h34m46.134605739s) [upload=12040]: Done starting upload Aug 7 21:14:11.619622676 (11h34m46.13472524s) [upload=12040]: Uploading segments... Aug 7 21:14:11.619678842 (11h34m46.134781443s) [upload=12040]: Got next segment. Inline: true Aug 7 21:14:11.619698988 (11h34m46.134801579s) [upload=12040]: Waiting for error group managing segments... Aug 7 21:14:11.619745714 (11h34m46.1348483s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:11.619750849 (11h34m46.134853438s) [upload=12040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:11.619754200 (11h34m46.134856779s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:11.587599715 (11h35m46.102702292s) [upload=12040]: Done uploading segments... Aug 7 21:15:11.587622036 (11h35m46.102724617s) [upload=12040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:11.944029087 (11h35m46.459131673s) [upload=12059]: Starting upload Aug 7 21:15:11.944115611 (11h35m46.459218173s) [upload=12059]: Done starting upload Aug 7 21:15:11.944188828 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000451952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.001256924s FAIL (2025-08-07 21:20:29 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001282905s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 30.000844366s FAIL (2025-08-07 21:20:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00091252s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [...................................................X.XXXXXXX] duration: 30.001093898s FAIL (2025-08-07 21:20:25 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8s) [upload=12010]: Done starting upload Aug 7 21:12:27.468432459 (11h33m1.983535027s) [upload=12010]: Uploading segments... Aug 7 21:12:27.468502812 (11h33m1.983605391s) [upload=12010]: Got next segment. Inline: true Aug 7 21:12:27.468509793 (11h33m1.983612369s) [upload=12010]: Waiting for error group managing segments... Aug 7 21:12:27.468577961 (11h33m1.983680537s) [upload=12010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:27.468583494 (11h33m1.98368607s) [upload=12010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:27.468587033 (11h33m1.983689609s) [upload=12010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:27.713163795 (11h33m2.228266382s) [upload=12010]: Done uploading segments... Aug 7 21:12:27.713184263 (11h33m2.228286844s) [upload=12010]: Upload finished. err: <nil> Aug 7 21:13:00.260488755 (11h33m34.775591337s) [upload=12021]: Starting upload Aug 7 21:13:00.260589735 (11h33m34.775692306s) [upload=12021]: Done starting upload Aug 7 21:13:00.260688806 (11h33m34.775791377s) [upload=12021]: Uploading segments... Aug 7 21:13:00.260755956 (11h33m34.775858537s) [upload=12021]: Got next segment. Inline: true Aug 7 21:13:00.260760943 (11h33m34.775863519s) [upload=12021]: Waiting for error group managing segments... Aug 7 21:13:00.260825909 (11h33m34.775928474s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:00.260831124 (11h33m34.775933703s) [upload=12021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:00.260834322 (11h33m34.7759369s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:30.231998317 (11h34m4.747100894s) [upload=12021]: Done uploading segments... Aug 7 21:13:30.232017085 (11h34m4.74711966s) [upload=12021]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:34.490529998 (11h34m9.005632564s) [upload=12029]: Starting upload Aug 7 21:13:34.490599665 (11h34m9.005702244s) [upload=12029]: Done starting upload Aug 7 21:13:34.490712022 (11h34m9.005814599s) [upload=12029]: Uploading segments... Aug 7 21:13:34.490822609 (11h34m9.005925188s) [upload=12029]: Got next segment. Inline: true Aug 7 21:13:34.490828860 (11h34m9.005931437s) [upload=12029]: Waiting for error group managing segments... Aug 7 21:13:34.490903638 (11h34m9.006006215s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:34.490908813 (11h34m9.006011379s) [upload=12029]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:34.490911892 (11h34m9.00601446s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:34.880488092 (11h34m9.39559068s) [upload=12029]: Done uploading segments... Aug 7 21:13:34.880508926 (11h34m9.395611515s) [upload=12029]: Upload finished. err: <nil> Aug 7 21:14:45.394850433 (11h35m19.909953011s) [upload=12049]: Starting upload Aug 7 21:14:45.394946608 (11h35m19.910049187s) [upload=12049]: Done starting upload Aug 7 21:14:45.395065277 (11h35m19.910167846s) [upload=12049]: Uploading segments... Aug 7 21:14:45.395104752 (11h35m19.910207337s) [upload=12049]: Got next segment. Inline: true Aug 7 21:14:45.395110169 (11h35m19.910212745s) [upload=12049]: Waiting for error group managing segments... Aug 7 21:14:45.395152641 (11h35m19.910255207s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.395157021 (11h35m19.910259598s) [upload=12049]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.395161401 (11h35m19.910263982s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.359522906 (11h35m49.874625488s) [upload=12049]: Done uploading segments... Aug 7 21:15:15.359549039 (11h35m49.874651622s) [upload=12049]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:53.553631580 (11h36m28.068734159s) [upload=12066]: Starting upload Aug 7 21:15:53.553693025 (11h36m28.068795593s) [upload=12066]: Done starting upload Aug 7 21:15:53.553757062 (11h36m28.06885963s) [upload=12066]: Uploading segments... Aug 7 21:15:53.553809335 (11h36m28.068911922s) [upload=12066]: Got next segment. Inline: true Aug 7 21:15:53.553813895 (11h36m28.068916471s) [upload=12066]: Waiting for error group managing segments... Aug 7 21:15:53.553863687 (11h36m28.068966263s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:53.553867873 (11h36m28.068970453s) [upload=12066]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:53.553871084 (11h36m28.068973661s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:23.520202496 (11h36m58.035305071s) [upload=12066]: Done uploading segments... Aug 7 21:16:23.520223689 (11h36m58.035326265s) [upload=12066]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:49.902511179 (11h37m24.417613746s) [upload=12081]: Starting upload Aug 7 21:16:49.902589681 (11h37m24.417692258s) [upload=12081]: Done starting upload Aug 7 21:16:49.902694562 (11h37m24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001117676s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.................................................X..XXXXXXXX] duration: 30.000607425s FAIL (2025-08-07 21:20:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000653008s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.001634304s FAIL (2025-08-07 21:20:18 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001677023s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................................X..XXXXX] duration: 30.001366647s FAIL (2025-08-07 21:20:12 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== gin upload segment. Aug 7 21:15:05.037810064 (11h35m39.552912651s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.037834987 (11h35m39.552937608s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.037910294 (11h35m39.553012873s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038363714 (11h35m39.553466302s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038429797 (11h35m39.553532386s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038728250 (11h35m39.553830831s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.039092766 (11h35m39.554195355s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.039689079 (11h35m39.554791667s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040315765 (11h35m39.555418329s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040942288 (11h35m39.556044876s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.041546451 (11h35m39.556649039s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042086243 (11h35m39.557188834s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042565982 (11h35m39.557668574s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043348608 (11h35m39.558451196s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043913089 (11h35m39.559015677s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.051646181 (11h35m39.566748771s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.177029777 (11h35m39.692132351s) [upload=12056]: Next returned nil segment Aug 7 21:15:05.177048224 (11h35m39.692150808s) [upload=12056]: Waiting for error group managing segments... Aug 7 21:15:05.181493366 (11h35m39.696595929s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182082323 (11h35m39.697184889s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182627665 (11h35m39.697730245s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183210775 (11h35m39.698313344s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183776547 (11h35m39.69887913s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184100393 (11h35m39.699202963s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184388388 (11h35m39.699490959s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185120907 (11h35m39.700223474s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185748114 (11h35m39.700850683s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces. Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments... Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments... Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments... Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments... Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments... Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded Duration: 30.00138788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 30.000601423s FAIL (2025-08-07 21:19:59 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000622501s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 1m0.000739935s FAIL (2025-08-07 21:19:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000768584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [....................................................X.XXXXXX] duration: 30.000732993s FAIL (2025-08-07 21:19:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :55.646725223 (11h32m30.161827806s) [upload=12003]: Done starting upload Aug 7 21:11:55.646793999 (11h32m30.161896566s) [upload=12003]: Uploading segments... Aug 7 21:11:55.646853254 (11h32m30.161955836s) [upload=12003]: Got next segment. Inline: true Aug 7 21:11:55.646858259 (11h32m30.161960835s) [upload=12003]: Waiting for error group managing segments... Aug 7 21:11:55.646908687 (11h32m30.162011263s) [upload=12003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:55.646914002 (11h32m30.162016583s) [upload=12003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:55.646918834 (11h32m30.162021413s) [upload=12003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:55.718218370 (11h32m30.233320937s) [upload=12003]: Done uploading segments... Aug 7 21:11:55.718234789 (11h32m30.233337359s) [upload=12003]: Upload finished. err: <nil> Aug 7 21:12:27.468174562 (11h33m1.983277139s) [upload=12010]: Starting upload Aug 7 21:12:27.468269218 (11h33m1.983371788s) [upload=12010]: Done starting upload Aug 7 21:12:27.468432459 (11h33m1.983535027s) [upload=12010]: Uploading segments... Aug 7 21:12:27.468502812 (11h33m1.983605391s) [upload=12010]: Got next segment. Inline: true Aug 7 21:12:27.468509793 (11h33m1.983612369s) [upload=12010]: Waiting for error group managing segments... Aug 7 21:12:27.468577961 (11h33m1.983680537s) [upload=12010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:27.468583494 (11h33m1.98368607s) [upload=12010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:27.468587033 (11h33m1.983689609s) [upload=12010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:27.713163795 (11h33m2.228266382s) [upload=12010]: Done uploading segments... Aug 7 21:12:27.713184263 (11h33m2.228286844s) [upload=12010]: Upload finished. err: <nil> Aug 7 21:13:00.260488755 (11h33m34.775591337s) [upload=12021]: Starting upload Aug 7 21:13:00.260589735 (11h33m34.775692306s) [upload=12021]: Done starting upload Aug 7 21:13:00.260688806 (11h33m34.775791377s) [upload=12021]: Uploading segments... Aug 7 21:13:00.260755956 (11h33m34.775858537s) [upload=12021]: Got next segment. Inline: true Aug 7 21:13:00.260760943 (11h33m34.775863519s) [upload=12021]: Waiting for error group managing segments... Aug 7 21:13:00.260825909 (11h33m34.775928474s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:00.260831124 (11h33m34.775933703s) [upload=12021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:00.260834322 (11h33m34.7759369s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:30.231998317 (11h34m4.747100894s) [upload=12021]: Done uploading segments... Aug 7 21:13:30.232017085 (11h34m4.74711966s) [upload=12021]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:34.490529998 (11h34m9.005632564s) [upload=12029]: Starting upload Aug 7 21:13:34.490599665 (11h34m9.005702244s) [upload=12029]: Done starting upload Aug 7 21:13:34.490712022 (11h34m9.005814599s) [upload=12029]: Uploading segments... Aug 7 21:13:34.490822609 (11h34m9.005925188s) [upload=12029]: Got next segment. Inline: true Aug 7 21:13:34.490828860 (11h34m9.005931437s) [upload=12029]: Waiting for error group managing segments... Aug 7 21:13:34.490903638 (11h34m9.006006215s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:34.490908813 (11h34m9.006011379s) [upload=12029]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:34.490911892 (11h34m9.00601446s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:34.880488092 (11h34m9.39559068s) [upload=12029]: Done uploading segments... Aug 7 21:13:34.880508926 (11h34m9.395611515s) [upload=12029]: Upload finished. err: <nil> Aug 7 21:14:45.394850433 (11h35m19.909953011s) [upload=12049]: Starting upload Aug 7 21:14:45.394946608 (11h35m19.910049187s) [upload=12049]: Done starting upload Aug 7 21:14:45.395065277 (11h35m19.910167846s) [upload=12049]: Uploading segments... Aug 7 21:14:45.395104752 (11h35m19.910207337s) [upload=12049]: Got next segment. Inline: true Aug 7 21:14:45.395110169 (11h35m19.910212745s) [upload=12049]: Waiting for error group managing segments... Aug 7 21:14:45.395152641 (11h35m19.910255207s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.395157021 (11h35m19.910259598s) [upload=12049]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.395161401 (11h35m19.910263982s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.359522906 (11h35m49.874625488s) [upload=12049]: Done uploading segments... Aug 7 21:15:15.359549039 (11h35m49.874651622s) [upload=12049]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:53.553631580 (11h36m28.068734159s) [upload=12066]: Starting upload Aug 7 21:15:53.553693025 (11h36m28.068795593s) [upload=12066]: Done starting upload Aug 7 21:15:53.553757062 (11h36m28.06885963s) [upload=12066]: Uploading segments... Aug 7 21:15:53.553809335 (11h36m28.068911922s) [upload=12066]: Got next segment. Inline: true Aug 7 21:15:53.553813895 (11h36m28.068916471s) [upload=12066]: Waiting for error group managing segments... Aug 7 21:15:53.553863687 (11h36m28.068966263s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:53.553867873 (11h36m28.068970453s) [upload=12066]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:53.553871084 (11h36m28.068973661s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:23.520202496 (11h36m58.035305071s) [upload=12066]: Done uploading segments... Aug 7 21:16:23.520223689 (11h36m58.035326265s) [upload=12066]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:49.902511179 (11h37m24.417613746s) [upload=12081]: Starting upload Aug 7 21:16:49.902589681 (11h37m24.417692258s) [upload=12081]: Done starting upload Aug 7 21:16:49.902694562 (11h37m24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000775139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.001794135s FAIL (2025-08-07 21:19:47 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001810994s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................X..XXXX] duration: 30.001227276s FAIL (2025-08-07 21:19:42 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== egments... Aug 7 21:14:53.975615592 (11h35m28.490718158s) [upload=12045]: Upload finished. err: context deadline exceeded Aug 7 21:15:04.087421335 (11h35m38.602523899s) [upload=12056]: Starting upload Aug 7 21:15:04.087495497 (11h35m38.60259809s) [upload=12056]: Done starting upload Aug 7 21:15:04.087846916 (11h35m38.602949489s) [upload=12056]: Uploading segments... Aug 7 21:15:04.087906907 (11h35m38.603009495s) [upload=12056]: Got next segment. Inline: false Aug 7 21:15:04.087944794 (11h35m38.603047373s) [upload=12056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:04.087950406 (11h35m38.60305299s) [upload=12056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:05.037685811 (11h35m39.552788422s) [seg_pos={0 0}, upload=12056]: Begin upload segment... Aug 7 21:15:05.037761820 (11h35m39.552864401s) [seg_pos={0 0}, upload=12056]: Done begin upload segment. Aug 7 21:15:05.037810064 (11h35m39.552912651s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.037834987 (11h35m39.552937608s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.037910294 (11h35m39.553012873s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038363714 (11h35m39.553466302s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038429797 (11h35m39.553532386s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038728250 (11h35m39.553830831s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.039092766 (11h35m39.554195355s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.039689079 (11h35m39.554791667s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040315765 (11h35m39.555418329s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040942288 (11h35m39.556044876s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.041546451 (11h35m39.556649039s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042086243 (11h35m39.557188834s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042565982 (11h35m39.557668574s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043348608 (11h35m39.558451196s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043913089 (11h35m39.559015677s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.051646181 (11h35m39.566748771s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.177029777 (11h35m39.692132351s) [upload=12056]: Next returned nil segment Aug 7 21:15:05.177048224 (11h35m39.692150808s) [upload=12056]: Waiting for error group managing segments... Aug 7 21:15:05.181493366 (11h35m39.696595929s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182082323 (11h35m39.697184889s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182627665 (11h35m39.697730245s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183210775 (11h35m39.698313344s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183776547 (11h35m39.69887913s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184100393 (11h35m39.699202963s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184388388 (11h35m39.699490959s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185120907 (11h35m39.700223474s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185748114 (11h35m39.700850683s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces. Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments... Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments... Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments... Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded Duration: 30.001240638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.000714166s FAIL (2025-08-07 21:19:31 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 10.114714791 (11h30m44.629817371s) [upload=11973]: Starting upload Aug 7 21:10:10.114793168 (11h30m44.629895735s) [upload=11973]: Done starting upload Aug 7 21:10:10.114888340 (11h30m44.629990904s) [upload=11973]: Uploading segments... Aug 7 21:10:10.114944378 (11h30m44.630046964s) [upload=11973]: Got next segment. Inline: true Aug 7 21:10:10.114951005 (11h30m44.630053591s) [upload=11973]: Waiting for error group managing segments... Aug 7 21:10:10.115003955 (11h30m44.630106535s) [upload=11973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:10.115010339 (11h30m44.630112927s) [upload=11973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:10.115014017 (11h30m44.630116599s) [upload=11973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:10.426003751 (11h30m44.941106327s) [upload=11973]: Done uploading segments... Aug 7 21:10:10.426023733 (11h30m44.941126312s) [upload=11973]: Upload finished. err: <nil> Aug 7 21:11:07.821695009 (11h31m42.3367976s) [upload=11989]: Starting upload Aug 7 21:11:07.821828021 (11h31m42.336930607s) [upload=11989]: Done starting upload Aug 7 21:11:07.821974268 (11h31m42.337076852s) [upload=11989]: Uploading segments... Aug 7 21:11:07.822051165 (11h31m42.33715375s) [upload=11989]: Got next segment. Inline: true Aug 7 21:11:07.822058710 (11h31m42.337161296s) [upload=11989]: Waiting for error group managing segments... Aug 7 21:11:07.822137123 (11h31m42.337239714s) [upload=11989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:07.822143163 (11h31m42.33724575s) [upload=11989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:07.822147451 (11h31m42.337250044s) [upload=11989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:07.880868203 (11h31m42.395970789s) [upload=11989]: Done uploading segments... Aug 7 21:11:07.880886229 (11h31m42.395988796s) [upload=11989]: Upload finished. err: <nil> Aug 7 21:11:30.967897030 (11h32m5.482999597s) [upload=11995]: Starting upload Aug 7 21:11:30.967969502 (11h32m5.483072083s) [upload=11995]: Done starting upload Aug 7 21:11:30.968040172 (11h32m5.483142743s) [upload=11995]: Uploading segments... Aug 7 21:11:30.968088031 (11h32m5.483190618s) [upload=11995]: Got next segment. Inline: true Aug 7 21:11:30.968092912 (11h32m5.483195492s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:11:30.968156155 (11h32m5.483258717s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:30.968160963 (11h32m5.483263526s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:30.968163832 (11h32m5.483266409s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.040679973 (11h32m5.555782542s) [upload=11995]: Done uploading segments... Aug 7 21:11:31.040697828 (11h32m5.555800393s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:12:44.577632884 (11h33m19.092735462s) [upload=12014]: Starting upload Aug 7 21:12:44.577718424 (11h33m19.092821007s) [upload=12014]: Done starting upload Aug 7 21:12:44.577778064 (11h33m19.09288064s) [upload=12014]: Uploading segments... Aug 7 21:12:44.577870249 (11h33m19.09297311s) [upload=12014]: Got next segment. Inline: true Aug 7 21:12:44.577878347 (11h33m19.09298093s) [upload=12014]: Waiting for error group managing segments... Aug 7 21:12:44.577949813 (11h33m19.093052392s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:44.577956047 (11h33m19.093058626s) [upload=12014]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:44.577963216 (11h33m19.093065803s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:44.741155623 (11h33m19.256258199s) [upload=12014]: Done uploading segments... Aug 7 21:12:44.741173325 (11h33m19.256275891s) [upload=12014]: Upload finished. err: <nil> Aug 7 21:13:22.079763163 (11h33m56.594865742s) [upload=12027]: Starting upload Aug 7 21:13:22.079827372 (11h33m56.594929946s) [upload=12027]: Done starting upload Aug 7 21:13:22.079907596 (11h33m56.595010178s) [upload=12027]: Uploading segments... Aug 7 21:13:22.079966866 (11h33m56.595069445s) [upload=12027]: Got next segment. Inline: true Aug 7 21:13:22.079973866 (11h33m56.595076443s) [upload=12027]: Waiting for error group managing segments... Aug 7 21:13:22.080019901 (11h33m56.595122472s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:22.080024370 (11h33m56.595126947s) [upload=12027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:22.080029157 (11h33m56.595131732s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:22.199851686 (11h33m56.714954269s) [upload=12027]: Done uploading segments... Aug 7 21:13:22.199871202 (11h33m56.714973784s) [upload=12027]: Upload finished. err: <nil> Aug 7 21:14:11.619394074 (11h34m46.134496661s) [upload=12040]: Starting upload Aug 7 21:14:11.619503144 (11h34m46.134605739s) [upload=12040]: Done starting upload Aug 7 21:14:11.619622676 (11h34m46.13472524s) [upload=12040]: Uploading segments... Aug 7 21:14:11.619678842 (11h34m46.134781443s) [upload=12040]: Got next segment. Inline: true Aug 7 21:14:11.619698988 (11h34m46.134801579s) [upload=12040]: Waiting for error group managing segments... Aug 7 21:14:11.619745714 (11h34m46.1348483s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:11.619750849 (11h34m46.134853438s) [upload=12040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:11.619754200 (11h34m46.134856779s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:11.587599715 (11h35m46.102702292s) [upload=12040]: Done uploading segments... Aug 7 21:15:11.587622036 (11h35m46.102724617s) [upload=12040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:11.944029087 (11h35m46.459131673s) [upload=12059]: Starting upload Aug 7 21:15:11.944115611 (11h35m46.459218173s) [upload=12059]: Done starting upload Aug 7 21:15:11.944188828 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000772318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.005711336s FAIL (2025-08-07 21:19:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.005776335s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..................................................X..XXXXXXX] duration: 30.000735881s FAIL (2025-08-07 21:19:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000758579s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 3m0.407533411s FAIL (2025-08-07 21:19:17 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 3m0.407556725s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 30.001046054s FAIL (2025-08-07 21:19:14 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001069796s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................X..XXX] duration: 4m8.107676131s FAIL (2025-08-07 21:19:12 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Aug 7 21:14:23.835156944 (11h34m58.350259514s) [upload=12032]: Upload finished. err: context deadline exceeded Aug 7 21:14:24.005677868 (11h34m58.52078043s) [upload=12045]: Starting upload Aug 7 21:14:24.005744046 (11h34m58.520846622s) [upload=12045]: Done starting upload Aug 7 21:14:24.006013012 (11h34m58.521115575s) [upload=12045]: Uploading segments... Aug 7 21:14:24.006062700 (11h34m58.521165282s) [upload=12045]: Got next segment. Inline: false Aug 7 21:14:24.006089531 (11h34m58.521192094s) [upload=12045]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:24.006093389 (11h34m58.521195951s) [upload=12045]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:53.975481666 (11h35m28.490584248s) [upload=12045]: Next segment err: context deadline exceeded Aug 7 21:14:53.975596874 (11h35m28.490699444s) [upload=12045]: Done uploading segments... Aug 7 21:14:53.975615592 (11h35m28.490718158s) [upload=12045]: Upload finished. err: context deadline exceeded Aug 7 21:15:04.087421335 (11h35m38.602523899s) [upload=12056]: Starting upload Aug 7 21:15:04.087495497 (11h35m38.60259809s) [upload=12056]: Done starting upload Aug 7 21:15:04.087846916 (11h35m38.602949489s) [upload=12056]: Uploading segments... Aug 7 21:15:04.087906907 (11h35m38.603009495s) [upload=12056]: Got next segment. Inline: false Aug 7 21:15:04.087944794 (11h35m38.603047373s) [upload=12056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:04.087950406 (11h35m38.60305299s) [upload=12056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:05.037685811 (11h35m39.552788422s) [seg_pos={0 0}, upload=12056]: Begin upload segment... Aug 7 21:15:05.037761820 (11h35m39.552864401s) [seg_pos={0 0}, upload=12056]: Done begin upload segment. Aug 7 21:15:05.037810064 (11h35m39.552912651s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.037834987 (11h35m39.552937608s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.037910294 (11h35m39.553012873s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038363714 (11h35m39.553466302s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038429797 (11h35m39.553532386s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.038728250 (11h35m39.553830831s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.039092766 (11h35m39.554195355s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.039689079 (11h35m39.554791667s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040315765 (11h35m39.555418329s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.040942288 (11h35m39.556044876s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.041546451 (11h35m39.556649039s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042086243 (11h35m39.557188834s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.042565982 (11h35m39.557668574s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043348608 (11h35m39.558451196s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.043913089 (11h35m39.559015677s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.051646181 (11h35m39.566748771s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Uploading piece... Aug 7 21:15:05.177029777 (11h35m39.692132351s) [upload=12056]: Next returned nil segment Aug 7 21:15:05.177048224 (11h35m39.692150808s) [upload=12056]: Waiting for error group managing segments... Aug 7 21:15:05.181493366 (11h35m39.696595929s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182082323 (11h35m39.697184889s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.182627665 (11h35m39.697730245s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183210775 (11h35m39.698313344s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.183776547 (11h35m39.69887913s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184100393 (11h35m39.699202963s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.184388388 (11h35m39.699490959s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185120907 (11h35m39.700223474s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.185748114 (11h35m39.700850683s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil> Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces. Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments... Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded Duration: 4m8.107707005s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 1m0.00064765s FAIL (2025-08-07 21:18:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000680473s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...................................................X..XXXXXX] duration: 30.001956457s FAIL (2025-08-07 21:18:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001974552s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 2m0.001246556s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:18:44 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.001262104s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................X.XXXXX] duration: 30.001480567s FAIL (2025-08-07 21:18:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 82]: Starting upload Aug 7 21:10:42.945741284 (11h31m17.460843863s) [upload=11982]: Done starting upload Aug 7 21:10:42.945850852 (11h31m17.46095342s) [upload=11982]: Uploading segments... Aug 7 21:10:42.945892389 (11h31m17.460994975s) [upload=11982]: Got next segment. Inline: true Aug 7 21:10:42.945897243 (11h31m17.460999819s) [upload=11982]: Waiting for error group managing segments... Aug 7 21:10:42.945957561 (11h31m17.461060137s) [upload=11982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:42.945962699 (11h31m17.461065276s) [upload=11982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:42.945965846 (11h31m17.461068421s) [upload=11982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:43.061691785 (11h31m17.576794373s) [upload=11982]: Done uploading segments... Aug 7 21:10:43.061710851 (11h31m17.576813425s) [upload=11982]: Upload finished. err: <nil> Aug 7 21:11:55.646673634 (11h32m30.161776203s) [upload=12003]: Starting upload Aug 7 21:11:55.646725223 (11h32m30.161827806s) [upload=12003]: Done starting upload Aug 7 21:11:55.646793999 (11h32m30.161896566s) [upload=12003]: Uploading segments... Aug 7 21:11:55.646853254 (11h32m30.161955836s) [upload=12003]: Got next segment. Inline: true Aug 7 21:11:55.646858259 (11h32m30.161960835s) [upload=12003]: Waiting for error group managing segments... Aug 7 21:11:55.646908687 (11h32m30.162011263s) [upload=12003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:55.646914002 (11h32m30.162016583s) [upload=12003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:55.646918834 (11h32m30.162021413s) [upload=12003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:55.718218370 (11h32m30.233320937s) [upload=12003]: Done uploading segments... Aug 7 21:11:55.718234789 (11h32m30.233337359s) [upload=12003]: Upload finished. err: <nil> Aug 7 21:12:27.468174562 (11h33m1.983277139s) [upload=12010]: Starting upload Aug 7 21:12:27.468269218 (11h33m1.983371788s) [upload=12010]: Done starting upload Aug 7 21:12:27.468432459 (11h33m1.983535027s) [upload=12010]: Uploading segments... Aug 7 21:12:27.468502812 (11h33m1.983605391s) [upload=12010]: Got next segment. Inline: true Aug 7 21:12:27.468509793 (11h33m1.983612369s) [upload=12010]: Waiting for error group managing segments... Aug 7 21:12:27.468577961 (11h33m1.983680537s) [upload=12010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:27.468583494 (11h33m1.98368607s) [upload=12010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:27.468587033 (11h33m1.983689609s) [upload=12010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:27.713163795 (11h33m2.228266382s) [upload=12010]: Done uploading segments... Aug 7 21:12:27.713184263 (11h33m2.228286844s) [upload=12010]: Upload finished. err: <nil> Aug 7 21:13:00.260488755 (11h33m34.775591337s) [upload=12021]: Starting upload Aug 7 21:13:00.260589735 (11h33m34.775692306s) [upload=12021]: Done starting upload Aug 7 21:13:00.260688806 (11h33m34.775791377s) [upload=12021]: Uploading segments... Aug 7 21:13:00.260755956 (11h33m34.775858537s) [upload=12021]: Got next segment. Inline: true Aug 7 21:13:00.260760943 (11h33m34.775863519s) [upload=12021]: Waiting for error group managing segments... Aug 7 21:13:00.260825909 (11h33m34.775928474s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:00.260831124 (11h33m34.775933703s) [upload=12021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:00.260834322 (11h33m34.7759369s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:30.231998317 (11h34m4.747100894s) [upload=12021]: Done uploading segments... Aug 7 21:13:30.232017085 (11h34m4.74711966s) [upload=12021]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:34.490529998 (11h34m9.005632564s) [upload=12029]: Starting upload Aug 7 21:13:34.490599665 (11h34m9.005702244s) [upload=12029]: Done starting upload Aug 7 21:13:34.490712022 (11h34m9.005814599s) [upload=12029]: Uploading segments... Aug 7 21:13:34.490822609 (11h34m9.005925188s) [upload=12029]: Got next segment. Inline: true Aug 7 21:13:34.490828860 (11h34m9.005931437s) [upload=12029]: Waiting for error group managing segments... Aug 7 21:13:34.490903638 (11h34m9.006006215s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:34.490908813 (11h34m9.006011379s) [upload=12029]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:34.490911892 (11h34m9.00601446s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:34.880488092 (11h34m9.39559068s) [upload=12029]: Done uploading segments... Aug 7 21:13:34.880508926 (11h34m9.395611515s) [upload=12029]: Upload finished. err: <nil> Aug 7 21:14:45.394850433 (11h35m19.909953011s) [upload=12049]: Starting upload Aug 7 21:14:45.394946608 (11h35m19.910049187s) [upload=12049]: Done starting upload Aug 7 21:14:45.395065277 (11h35m19.910167846s) [upload=12049]: Uploading segments... Aug 7 21:14:45.395104752 (11h35m19.910207337s) [upload=12049]: Got next segment. Inline: true Aug 7 21:14:45.395110169 (11h35m19.910212745s) [upload=12049]: Waiting for error group managing segments... Aug 7 21:14:45.395152641 (11h35m19.910255207s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.395157021 (11h35m19.910259598s) [upload=12049]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.395161401 (11h35m19.910263982s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.359522906 (11h35m49.874625488s) [upload=12049]: Done uploading segments... Aug 7 21:15:15.359549039 (11h35m49.874651622s) [upload=12049]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:53.553631580 (11h36m28.068734159s) [upload=12066]: Starting upload Aug 7 21:15:53.553693025 (11h36m28.068795593s) [upload=12066]: Done starting upload Aug 7 21:15:53.553757062 (11h36m28.06885963s) [upload=12066]: Uploading segments... Aug 7 21:15:53.553809335 (11h36m28.068911922s) [upload=12066]: Got next segment. Inline: true Aug 7 21:15:53.553813895 (11h36m28.068916471s) [upload=12066]: Waiting for error group managing segments... Aug 7 21:15:53.553863687 (11h36m28.068966263s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:53.553867873 (11h36m28.068970453s) [upload=12066]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:53.553871084 (11h36m28.068973661s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:23.520202496 (11h36m58.035305071s) [upload=12066]: Done uploading segments... Aug 7 21:16:23.520223689 (11h36m58.035326265s) [upload=12066]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:49.902511179 (11h37m24.417613746s) [upload=12081]: Starting upload Aug 7 21:16:49.902589681 (11h37m24.417692258s) [upload=12081]: Done starting upload Aug 7 21:16:49.902694562 (11h37m24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001511219s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.004714891s FAIL (2025-08-07 21:18:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.004734149s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.000709584s FAIL (2025-08-07 21:18:30 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== <nil> Aug 7 21:09:51.381222521 (11h30m25.896325102s) [upload=11966]: Starting upload Aug 7 21:09:51.381312517 (11h30m25.896415103s) [upload=11966]: Done starting upload Aug 7 21:09:51.381397386 (11h30m25.896499962s) [upload=11966]: Uploading segments... Aug 7 21:09:51.381448522 (11h30m25.896551107s) [upload=11966]: Got next segment. Inline: true Aug 7 21:09:51.381454533 (11h30m25.896557114s) [upload=11966]: Waiting for error group managing segments... Aug 7 21:09:51.381512553 (11h30m25.896615135s) [upload=11966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:51.381518068 (11h30m25.89662065s) [upload=11966]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:51.381521548 (11h30m25.896624127s) [upload=11966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:51.448002347 (11h30m25.963104916s) [upload=11966]: Done uploading segments... Aug 7 21:09:51.448032497 (11h30m25.963135063s) [upload=11966]: Upload finished. err: <nil> Aug 7 21:10:10.114714791 (11h30m44.629817371s) [upload=11973]: Starting upload Aug 7 21:10:10.114793168 (11h30m44.629895735s) [upload=11973]: Done starting upload Aug 7 21:10:10.114888340 (11h30m44.629990904s) [upload=11973]: Uploading segments... Aug 7 21:10:10.114944378 (11h30m44.630046964s) [upload=11973]: Got next segment. Inline: true Aug 7 21:10:10.114951005 (11h30m44.630053591s) [upload=11973]: Waiting for error group managing segments... Aug 7 21:10:10.115003955 (11h30m44.630106535s) [upload=11973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:10.115010339 (11h30m44.630112927s) [upload=11973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:10.115014017 (11h30m44.630116599s) [upload=11973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:10.426003751 (11h30m44.941106327s) [upload=11973]: Done uploading segments... Aug 7 21:10:10.426023733 (11h30m44.941126312s) [upload=11973]: Upload finished. err: <nil> Aug 7 21:11:07.821695009 (11h31m42.3367976s) [upload=11989]: Starting upload Aug 7 21:11:07.821828021 (11h31m42.336930607s) [upload=11989]: Done starting upload Aug 7 21:11:07.821974268 (11h31m42.337076852s) [upload=11989]: Uploading segments... Aug 7 21:11:07.822051165 (11h31m42.33715375s) [upload=11989]: Got next segment. Inline: true Aug 7 21:11:07.822058710 (11h31m42.337161296s) [upload=11989]: Waiting for error group managing segments... Aug 7 21:11:07.822137123 (11h31m42.337239714s) [upload=11989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:07.822143163 (11h31m42.33724575s) [upload=11989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:07.822147451 (11h31m42.337250044s) [upload=11989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:07.880868203 (11h31m42.395970789s) [upload=11989]: Done uploading segments... Aug 7 21:11:07.880886229 (11h31m42.395988796s) [upload=11989]: Upload finished. err: <nil> Aug 7 21:11:30.967897030 (11h32m5.482999597s) [upload=11995]: Starting upload Aug 7 21:11:30.967969502 (11h32m5.483072083s) [upload=11995]: Done starting upload Aug 7 21:11:30.968040172 (11h32m5.483142743s) [upload=11995]: Uploading segments... Aug 7 21:11:30.968088031 (11h32m5.483190618s) [upload=11995]: Got next segment. Inline: true Aug 7 21:11:30.968092912 (11h32m5.483195492s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:11:30.968156155 (11h32m5.483258717s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:30.968160963 (11h32m5.483263526s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:30.968163832 (11h32m5.483266409s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.040679973 (11h32m5.555782542s) [upload=11995]: Done uploading segments... Aug 7 21:11:31.040697828 (11h32m5.555800393s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:12:44.577632884 (11h33m19.092735462s) [upload=12014]: Starting upload Aug 7 21:12:44.577718424 (11h33m19.092821007s) [upload=12014]: Done starting upload Aug 7 21:12:44.577778064 (11h33m19.09288064s) [upload=12014]: Uploading segments... Aug 7 21:12:44.577870249 (11h33m19.09297311s) [upload=12014]: Got next segment. Inline: true Aug 7 21:12:44.577878347 (11h33m19.09298093s) [upload=12014]: Waiting for error group managing segments... Aug 7 21:12:44.577949813 (11h33m19.093052392s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:44.577956047 (11h33m19.093058626s) [upload=12014]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:44.577963216 (11h33m19.093065803s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:44.741155623 (11h33m19.256258199s) [upload=12014]: Done uploading segments... Aug 7 21:12:44.741173325 (11h33m19.256275891s) [upload=12014]: Upload finished. err: <nil> Aug 7 21:13:22.079763163 (11h33m56.594865742s) [upload=12027]: Starting upload Aug 7 21:13:22.079827372 (11h33m56.594929946s) [upload=12027]: Done starting upload Aug 7 21:13:22.079907596 (11h33m56.595010178s) [upload=12027]: Uploading segments... Aug 7 21:13:22.079966866 (11h33m56.595069445s) [upload=12027]: Got next segment. Inline: true Aug 7 21:13:22.079973866 (11h33m56.595076443s) [upload=12027]: Waiting for error group managing segments... Aug 7 21:13:22.080019901 (11h33m56.595122472s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:22.080024370 (11h33m56.595126947s) [upload=12027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:22.080029157 (11h33m56.595131732s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:22.199851686 (11h33m56.714954269s) [upload=12027]: Done uploading segments... Aug 7 21:13:22.199871202 (11h33m56.714973784s) [upload=12027]: Upload finished. err: <nil> Aug 7 21:14:11.619394074 (11h34m46.134496661s) [upload=12040]: Starting upload Aug 7 21:14:11.619503144 (11h34m46.134605739s) [upload=12040]: Done starting upload Aug 7 21:14:11.619622676 (11h34m46.13472524s) [upload=12040]: Uploading segments... Aug 7 21:14:11.619678842 (11h34m46.134781443s) [upload=12040]: Got next segment. Inline: true Aug 7 21:14:11.619698988 (11h34m46.134801579s) [upload=12040]: Waiting for error group managing segments... Aug 7 21:14:11.619745714 (11h34m46.1348483s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:11.619750849 (11h34m46.134853438s) [upload=12040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:11.619754200 (11h34m46.134856779s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:11.587599715 (11h35m46.102702292s) [upload=12040]: Done uploading segments... Aug 7 21:15:11.587622036 (11h35m46.102724617s) [upload=12040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:11.944029087 (11h35m46.459131673s) [upload=12059]: Starting upload Aug 7 21:15:11.944115611 (11h35m46.459218173s) [upload=12059]: Done starting upload Aug 7 21:15:11.944188828 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000732443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 30.001638938s FAIL (2025-08-07 21:18:27 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001666156s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................................................X..XXXXX] duration: 30.000742719s FAIL (2025-08-07 21:18:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000762392s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................................X.XXXX] duration: 30.000609759s FAIL (2025-08-07 21:18:12 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11h30m33.976458859s) [upload=11970]: Starting upload Aug 7 21:09:59.461514228 (11h30m33.976616814s) [upload=11970]: Done starting upload Aug 7 21:09:59.461657967 (11h30m33.976760542s) [upload=11970]: Uploading segments... Aug 7 21:09:59.461728003 (11h30m33.976830589s) [upload=11970]: Got next segment. Inline: true Aug 7 21:09:59.461733779 (11h30m33.976836356s) [upload=11970]: Waiting for error group managing segments... Aug 7 21:09:59.461777635 (11h30m33.976880202s) [upload=11970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:59.461783114 (11h30m33.97688569s) [upload=11970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:59.461787951 (11h30m33.976890529s) [upload=11970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:59.731460482 (11h30m34.246563058s) [upload=11970]: Done uploading segments... Aug 7 21:09:59.731478076 (11h30m34.246580652s) [upload=11970]: Upload finished. err: <nil> Aug 7 21:10:42.945656363 (11h31m17.460758936s) [upload=11982]: Starting upload Aug 7 21:10:42.945741284 (11h31m17.460843863s) [upload=11982]: Done starting upload Aug 7 21:10:42.945850852 (11h31m17.46095342s) [upload=11982]: Uploading segments... Aug 7 21:10:42.945892389 (11h31m17.460994975s) [upload=11982]: Got next segment. Inline: true Aug 7 21:10:42.945897243 (11h31m17.460999819s) [upload=11982]: Waiting for error group managing segments... Aug 7 21:10:42.945957561 (11h31m17.461060137s) [upload=11982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:42.945962699 (11h31m17.461065276s) [upload=11982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:42.945965846 (11h31m17.461068421s) [upload=11982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:43.061691785 (11h31m17.576794373s) [upload=11982]: Done uploading segments... Aug 7 21:10:43.061710851 (11h31m17.576813425s) [upload=11982]: Upload finished. err: <nil> Aug 7 21:11:55.646673634 (11h32m30.161776203s) [upload=12003]: Starting upload Aug 7 21:11:55.646725223 (11h32m30.161827806s) [upload=12003]: Done starting upload Aug 7 21:11:55.646793999 (11h32m30.161896566s) [upload=12003]: Uploading segments... Aug 7 21:11:55.646853254 (11h32m30.161955836s) [upload=12003]: Got next segment. Inline: true Aug 7 21:11:55.646858259 (11h32m30.161960835s) [upload=12003]: Waiting for error group managing segments... Aug 7 21:11:55.646908687 (11h32m30.162011263s) [upload=12003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:55.646914002 (11h32m30.162016583s) [upload=12003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:55.646918834 (11h32m30.162021413s) [upload=12003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:55.718218370 (11h32m30.233320937s) [upload=12003]: Done uploading segments... Aug 7 21:11:55.718234789 (11h32m30.233337359s) [upload=12003]: Upload finished. err: <nil> Aug 7 21:12:27.468174562 (11h33m1.983277139s) [upload=12010]: Starting upload Aug 7 21:12:27.468269218 (11h33m1.983371788s) [upload=12010]: Done starting upload Aug 7 21:12:27.468432459 (11h33m1.983535027s) [upload=12010]: Uploading segments... Aug 7 21:12:27.468502812 (11h33m1.983605391s) [upload=12010]: Got next segment. Inline: true Aug 7 21:12:27.468509793 (11h33m1.983612369s) [upload=12010]: Waiting for error group managing segments... Aug 7 21:12:27.468577961 (11h33m1.983680537s) [upload=12010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:27.468583494 (11h33m1.98368607s) [upload=12010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:27.468587033 (11h33m1.983689609s) [upload=12010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:27.713163795 (11h33m2.228266382s) [upload=12010]: Done uploading segments... Aug 7 21:12:27.713184263 (11h33m2.228286844s) [upload=12010]: Upload finished. err: <nil> Aug 7 21:13:00.260488755 (11h33m34.775591337s) [upload=12021]: Starting upload Aug 7 21:13:00.260589735 (11h33m34.775692306s) [upload=12021]: Done starting upload Aug 7 21:13:00.260688806 (11h33m34.775791377s) [upload=12021]: Uploading segments... Aug 7 21:13:00.260755956 (11h33m34.775858537s) [upload=12021]: Got next segment. Inline: true Aug 7 21:13:00.260760943 (11h33m34.775863519s) [upload=12021]: Waiting for error group managing segments... Aug 7 21:13:00.260825909 (11h33m34.775928474s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:00.260831124 (11h33m34.775933703s) [upload=12021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:00.260834322 (11h33m34.7759369s) [upload=12021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:30.231998317 (11h34m4.747100894s) [upload=12021]: Done uploading segments... Aug 7 21:13:30.232017085 (11h34m4.74711966s) [upload=12021]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:34.490529998 (11h34m9.005632564s) [upload=12029]: Starting upload Aug 7 21:13:34.490599665 (11h34m9.005702244s) [upload=12029]: Done starting upload Aug 7 21:13:34.490712022 (11h34m9.005814599s) [upload=12029]: Uploading segments... Aug 7 21:13:34.490822609 (11h34m9.005925188s) [upload=12029]: Got next segment. Inline: true Aug 7 21:13:34.490828860 (11h34m9.005931437s) [upload=12029]: Waiting for error group managing segments... Aug 7 21:13:34.490903638 (11h34m9.006006215s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:34.490908813 (11h34m9.006011379s) [upload=12029]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:34.490911892 (11h34m9.00601446s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:34.880488092 (11h34m9.39559068s) [upload=12029]: Done uploading segments... Aug 7 21:13:34.880508926 (11h34m9.395611515s) [upload=12029]: Upload finished. err: <nil> Aug 7 21:14:45.394850433 (11h35m19.909953011s) [upload=12049]: Starting upload Aug 7 21:14:45.394946608 (11h35m19.910049187s) [upload=12049]: Done starting upload Aug 7 21:14:45.395065277 (11h35m19.910167846s) [upload=12049]: Uploading segments... Aug 7 21:14:45.395104752 (11h35m19.910207337s) [upload=12049]: Got next segment. Inline: true Aug 7 21:14:45.395110169 (11h35m19.910212745s) [upload=12049]: Waiting for error group managing segments... Aug 7 21:14:45.395152641 (11h35m19.910255207s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.395157021 (11h35m19.910259598s) [upload=12049]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.395161401 (11h35m19.910263982s) [upload=12049]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.359522906 (11h35m49.874625488s) [upload=12049]: Done uploading segments... Aug 7 21:15:15.359549039 (11h35m49.874651622s) [upload=12049]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:53.553631580 (11h36m28.068734159s) [upload=12066]: Starting upload Aug 7 21:15:53.553693025 (11h36m28.068795593s) [upload=12066]: Done starting upload Aug 7 21:15:53.553757062 (11h36m28.06885963s) [upload=12066]: Uploading segments... Aug 7 21:15:53.553809335 (11h36m28.068911922s) [upload=12066]: Got next segment. Inline: true Aug 7 21:15:53.553813895 (11h36m28.068916471s) [upload=12066]: Waiting for error group managing segments... Aug 7 21:15:53.553863687 (11h36m28.068966263s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:53.553867873 (11h36m28.068970453s) [upload=12066]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:53.553871084 (11h36m28.068973661s) [upload=12066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:23.520202496 (11h36m58.035305071s) [upload=12066]: Done uploading segments... Aug 7 21:16:23.520223689 (11h36m58.035326265s) [upload=12066]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:49.902511179 (11h37m24.417613746s) [upload=12081]: Starting upload Aug 7 21:16:49.902589681 (11h37m24.417692258s) [upload=12081]: Done starting upload Aug 7 21:16:49.902694562 (11h37m24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000635327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.002128014s FAIL (2025-08-07 21:18:05 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== oad=11997]: Uploading piece... Aug 7 21:11:49.974975266 (11h32m24.490077832s) [upload=11997]: Next returned nil segment Aug 7 21:11:49.974994144 (11h32m24.49009672s) [upload=11997]: Waiting for error group managing segments... Aug 7 21:11:49.988542190 (11h32m24.50364476s) [noise=true , address=104.36.82.58:4266 , piece_id=F5H6WPF4UQZKLTWOF6IXSHMCHHTQCM6JMVOO75IBFZWV4MTZPQYQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:49.995910724 (11h32m24.511013304s) [noise=true , address=104.36.82.8:4996 , piece_id=SDSB2I6IRVHIKYNCTXXMMS7QRHP6WQJLTGVSEKHM7TZJXNMSPOTA, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:49.996610690 (11h32m24.511713266s) [noise=true , address=45.91.116.5:4322 , piece_id=N2NGYY7VH5K7XORMB263LFAEA6QAG34XIX2EEHA5VRVWN6Y7ICOQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:49.999701015 (11h32m24.514803592s) [noise=true , address=104.36.82.61:4405 , piece_id=IOUNUAQUDRNNPKNFDRBIAM4MHCLESO4JHH3YGXYPRQDHN4IMSF7Q, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.008965761 (11h32m24.524068325s) [noise=true , address=104.36.82.60:4353 , piece_id=ULO2AG7LYJZQW6RPJ7MELFSAXVO6PGJOQEY74DD54UKFN62PVBPA, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.017145206 (11h32m24.532247782s) [noise=true , address=104.36.82.38:4693 , piece_id=VMPINEP4N2EW23O73YZDTLJOK5KTAX7IURQ36YCOL4LXWHAYVNUQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.022781027 (11h32m24.537883606s) [noise=true , address=104.36.82.16:4302 , piece_id=TXFXBAD3J4MFZGTEGPBN77H64Z3NCHCTJOJ23UJVWDRDH5FZUZQA, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.023398509 (11h32m24.53850109s) [noise=true , address=104.36.82.14:4256 , piece_id=QOALWOORJKGIRA7SFLFOCDAQZXEUOUIWV3EXABX2ANHJCPUAOVMQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.043528291 (11h32m24.558630869s) [noise=true , address=104.36.82.5:4847 , piece_id=ALFYQ5V3F74HBJW6Y4EEZQ5CPH2FK6TMVZN24M5N26FJUPTKPBBQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.046437033 (11h32m24.561539599s) [noise=true , address=104.36.82.17:4333 , piece_id=RH3RNSP2MCJ55QOLMLM2GG6XIQYMUWDT5CIL4IIUVW6TSBKQ4YVQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.087365961 (11h32m24.602468531s) [noise=true , address=104.36.82.47:4550 , piece_id=XZLYNUXHURQTA5ZV3QM44OQFELJJ26UUSPWH4HXXSK5GNZ4ZNHTA, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.160388869 (11h32m24.675491445s) [noise=true , address=104.36.82.52:4095 , piece_id=TIUS4WMCTIOX37O3FTYAFPI6QPGEIWCQ5G2B53O6KXA73W2JWVDA, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.301621081 (11h32m24.816723645s) [noise=true , address=104.36.82.2:4737 , piece_id=L2KDBV6RNEBV6EWSQLN4XX44BA3PMBXGWNU3BNXTMGF26TZMKKLQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.466287316 (11h32m24.981389887s) [noise=true , address=79.127.205.245:4228 , piece_id=SNJK7UVAPUBB4GRL5P4RQ7G7IQ45YISRD7LTTTH47MACDQKOGVXQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: <nil> Aug 7 21:11:50.466310392 (11h32m24.981412969s) [seg_pos={0 0}, upload=11997]: Segment reached optimal threshold of 14 pieces. Aug 7 21:11:50.466545670 (11h32m24.981648233s) [noise=true , address=104.36.82.23:4473 , piece_id=TUI3ZSQYKBCVF7BIQ4EI6ZDURYOTCF2HA3TRI3OGR376XMSCPBTQ, seg_pos={0 0}, upload=11997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.466997931 (11h32m24.982100496s) [noise=true , address=79.127.205.234:4598 , piece_id=QKLRPLUZPQDQYDDT3JDYNQVI3OJ3S4JJQ7GHM5ZEX7DJNH7COWRA, seg_pos={0 0}, upload=11997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1em3M1WGGYB3mGai2ayAnwWFQU3c6LA3MJQf3PekZcvssk5qaT): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.467162262 (11h32m24.982264825s) [noise=true , address=64.49.13.170:4722 , piece_id=NG3CUNOYSNNKBPRWBKICIWMVFVT6Z7S3QMLDWBL3OXRYK37GNSDA, seg_pos={0 0}, upload=11997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PwEo82z9nhp4yVv7Ff55AwxHgRLhfuSMmTghTimaoAHu76PXf): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.467246223 (11h32m24.982348803s) [noise=true , address=104.36.82.44:4880 , piece_id=EPT4SHLU25E6NWHUDABBKFDG3C3YVFPIF5I35AJI4FZVKIIUED5Q, seg_pos={0 0}, upload=11997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF Aug 7 21:11:50.467281805 (11h32m24.982384369s) [seg_pos={0 0}, upload=11997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:11:50.467362207 (11h32m24.982464787s) [upload=11997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:11:50.467367803 (11h32m24.982470386s) [upload=11997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:57.502219885 (11h33m32.017322462s) [upload=11997]: Done uploading segments... Aug 7 21:12:57.502247705 (11h33m32.017350283s) [upload=11997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:12:57.635219958 (11h33m32.150322528s) [upload=12020]: Starting upload Aug 7 21:12:57.635291431 (11h33m32.150394007s) [upload=12020]: Done starting upload Aug 7 21:12:57.635412367 (11h33m32.150514934s) [upload=12020]: Uploading segments... Aug 7 21:12:57.635706561 (11h33m32.150809138s) [upload=12020]: Got next segment. Inline: false Aug 7 21:12:57.635807146 (11h33m32.150909713s) [upload=12020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:57.635812366 (11h33m32.150914942s) [upload=12020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:13:57.598803233 (11h34m32.113905819s) [upload=12020]: Next segment err: context deadline exceeded Aug 7 21:13:57.598926914 (11h34m32.114029476s) [upload=12020]: Done uploading segments... Aug 7 21:13:57.598944574 (11h34m32.114047136s) [upload=12020]: Upload finished. err: context deadline exceeded Aug 7 21:13:57.707893490 (11h34m32.222996072s) [upload=12034]: Starting upload Aug 7 21:13:57.708008457 (11h34m32.223111035s) [upload=12034]: Done starting upload Aug 7 21:13:57.708257239 (11h34m32.223359807s) [upload=12034]: Uploading segments... Aug 7 21:13:57.708628569 (11h34m32.223731149s) [upload=12034]: Got next segment. Inline: false Aug 7 21:13:57.708697917 (11h34m32.2238005s) [upload=12034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:57.708703428 (11h34m32.223806004s) [upload=12034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:57.677372272 (11h35m32.19247486s) [upload=12034]: Next segment err: context deadline exceeded Aug 7 21:14:57.677532945 (11h35m32.192635511s) [upload=12034]: Done uploading segments... Aug 7 21:14:57.677558723 (11h35m32.192661287s) [upload=12034]: Upload finished. err: context deadline exceeded Aug 7 21:14:57.906428111 (11h35m32.421530675s) [upload=12052]: Starting upload Aug 7 21:14:57.906515646 (11h35m32.421618231s) [upload=12052]: Done starting upload Aug 7 21:14:57.906583348 (11h35m32.421685918s) [upload=12052]: Uploading segments... Aug 7 21:14:57.907007505 (11h35m32.422110088s) [upload=12052]: Got next segment. Inline: false Aug 7 21:14:57.907095500 (11h35m32.42219808s) [upload=12052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:57.907102983 (11h35m32.422205563s) [upload=12052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:57.857675330 (11h36m32.372777915s) [upload=12052]: Next segment err: context deadline exceeded Aug 7 21:15:57.857827342 (11h36m32.372929919s) [upload=12052]: Done uploading segments... Aug 7 21:15:57.857845992 (11h36m32.372948573s) [upload=12052]: Upload finished. err: context deadline exceeded Aug 7 21:16:05.804869311 (11h36m40.319971891s) [upload=12067]: Starting upload Aug 7 21:16:05.804961392 (11h36m40.32006397s) [upload=12067]: Done starting upload Aug 7 21:16:05.805366037 (11h36m40.320468613s) [upload=12067]: Uploading segments... Aug 7 21:16:05.805444950 (11h36m40.320547537s) [upload=12067]: Got next segment. Inline: false Aug 7 21:16:05.805524445 (11h36m40.320627032s) [upload=12067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:05.805530950 (11h36m40.320633522s) [upload=12067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:05.764865199 (11h37m40.279967765s) [upload=12067]: Next segment err: context deadline exceeded Aug 7 21:17:05.764907057 (11h37m40.280009637s) [upload=12067]: Done uploading segments... Aug 7 21:17:05.764922340 (11h37m40.280024907s) [upload=12067]: Upload finished. err: context deadline exceeded Aug 7 21:17:05.881294192 (11h37m40.396396757s) [upload=12086]: Starting upload Aug 7 21:17:05.881386085 (11h37m40.396488812s) [upload=12086]: Done starting upload Aug 7 21:17:05.881507614 (11h37m40.396610181s) [upload=12086]: Uploading segments... Aug 7 21:17:05.881832218 (11h37m40.396934788s) [upload=12086]: Got next segment. Inline: false Aug 7 21:17:05.881894583 (11h37m40.396997165s) [upload=12086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:05.881902816 (11h37m40.397005405s) [upload=12086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:05.851357904 (11h38m40.366460568s) [upload=12086]: Next segment err: context deadline exceeded Aug 7 21:18:05.851380345 (11h38m40.366482922s) [upload=12086]: Done uploading segments... Aug 7 21:18:05.851399154 (11h38m40.366501726s) [upload=12086]: Upload finished. err: context deadline exceeded Duration: 1m0.002150594s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.000894344s FAIL (2025-08-07 21:17:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000936357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.000697063s FAIL (2025-08-07 21:17:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000724946s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................X..XXXX] duration: 30.001390383s FAIL (2025-08-07 21:17:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001524031s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.000662364s FAIL (2025-08-07 21:17:23 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=11949]: Upload finished. err: <nil> Aug 7 21:09:28.927883911 (11h30m3.442986491s) [upload=11961]: Starting upload Aug 7 21:09:28.927978534 (11h30m3.44308112s) [upload=11961]: Done starting upload Aug 7 21:09:28.928091710 (11h30m3.443194288s) [upload=11961]: Uploading segments... Aug 7 21:09:28.928200886 (11h30m3.443303473s) [upload=11961]: Got next segment. Inline: true Aug 7 21:09:28.928228777 (11h30m3.443331364s) [upload=11961]: Waiting for error group managing segments... Aug 7 21:09:28.928494113 (11h30m3.443596694s) [upload=11961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:28.928504795 (11h30m3.443607384s) [upload=11961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:28.928508406 (11h30m3.443610992s) [upload=11961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:28.988058639 (11h30m3.503161225s) [upload=11961]: Done uploading segments... Aug 7 21:09:28.988080739 (11h30m3.503183322s) [upload=11961]: Upload finished. err: <nil> Aug 7 21:09:51.381222521 (11h30m25.896325102s) [upload=11966]: Starting upload Aug 7 21:09:51.381312517 (11h30m25.896415103s) [upload=11966]: Done starting upload Aug 7 21:09:51.381397386 (11h30m25.896499962s) [upload=11966]: Uploading segments... Aug 7 21:09:51.381448522 (11h30m25.896551107s) [upload=11966]: Got next segment. Inline: true Aug 7 21:09:51.381454533 (11h30m25.896557114s) [upload=11966]: Waiting for error group managing segments... Aug 7 21:09:51.381512553 (11h30m25.896615135s) [upload=11966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:09:51.381518068 (11h30m25.89662065s) [upload=11966]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:51.381521548 (11h30m25.896624127s) [upload=11966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:09:51.448002347 (11h30m25.963104916s) [upload=11966]: Done uploading segments... Aug 7 21:09:51.448032497 (11h30m25.963135063s) [upload=11966]: Upload finished. err: <nil> Aug 7 21:10:10.114714791 (11h30m44.629817371s) [upload=11973]: Starting upload Aug 7 21:10:10.114793168 (11h30m44.629895735s) [upload=11973]: Done starting upload Aug 7 21:10:10.114888340 (11h30m44.629990904s) [upload=11973]: Uploading segments... Aug 7 21:10:10.114944378 (11h30m44.630046964s) [upload=11973]: Got next segment. Inline: true Aug 7 21:10:10.114951005 (11h30m44.630053591s) [upload=11973]: Waiting for error group managing segments... Aug 7 21:10:10.115003955 (11h30m44.630106535s) [upload=11973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:10.115010339 (11h30m44.630112927s) [upload=11973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:10.115014017 (11h30m44.630116599s) [upload=11973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:10.426003751 (11h30m44.941106327s) [upload=11973]: Done uploading segments... Aug 7 21:10:10.426023733 (11h30m44.941126312s) [upload=11973]: Upload finished. err: <nil> Aug 7 21:11:07.821695009 (11h31m42.3367976s) [upload=11989]: Starting upload Aug 7 21:11:07.821828021 (11h31m42.336930607s) [upload=11989]: Done starting upload Aug 7 21:11:07.821974268 (11h31m42.337076852s) [upload=11989]: Uploading segments... Aug 7 21:11:07.822051165 (11h31m42.33715375s) [upload=11989]: Got next segment. Inline: true Aug 7 21:11:07.822058710 (11h31m42.337161296s) [upload=11989]: Waiting for error group managing segments... Aug 7 21:11:07.822137123 (11h31m42.337239714s) [upload=11989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:07.822143163 (11h31m42.33724575s) [upload=11989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:07.822147451 (11h31m42.337250044s) [upload=11989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:07.880868203 (11h31m42.395970789s) [upload=11989]: Done uploading segments... Aug 7 21:11:07.880886229 (11h31m42.395988796s) [upload=11989]: Upload finished. err: <nil> Aug 7 21:11:30.967897030 (11h32m5.482999597s) [upload=11995]: Starting upload Aug 7 21:11:30.967969502 (11h32m5.483072083s) [upload=11995]: Done starting upload Aug 7 21:11:30.968040172 (11h32m5.483142743s) [upload=11995]: Uploading segments... Aug 7 21:11:30.968088031 (11h32m5.483190618s) [upload=11995]: Got next segment. Inline: true Aug 7 21:11:30.968092912 (11h32m5.483195492s) [upload=11995]: Waiting for error group managing segments... Aug 7 21:11:30.968156155 (11h32m5.483258717s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:30.968160963 (11h32m5.483263526s) [upload=11995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:30.968163832 (11h32m5.483266409s) [upload=11995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.040679973 (11h32m5.555782542s) [upload=11995]: Done uploading segments... Aug 7 21:11:31.040697828 (11h32m5.555800393s) [upload=11995]: Upload finished. err: <nil> Aug 7 21:12:44.577632884 (11h33m19.092735462s) [upload=12014]: Starting upload Aug 7 21:12:44.577718424 (11h33m19.092821007s) [upload=12014]: Done starting upload Aug 7 21:12:44.577778064 (11h33m19.09288064s) [upload=12014]: Uploading segments... Aug 7 21:12:44.577870249 (11h33m19.09297311s) [upload=12014]: Got next segment. Inline: true Aug 7 21:12:44.577878347 (11h33m19.09298093s) [upload=12014]: Waiting for error group managing segments... Aug 7 21:12:44.577949813 (11h33m19.093052392s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:44.577956047 (11h33m19.093058626s) [upload=12014]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:44.577963216 (11h33m19.093065803s) [upload=12014]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:44.741155623 (11h33m19.256258199s) [upload=12014]: Done uploading segments... Aug 7 21:12:44.741173325 (11h33m19.256275891s) [upload=12014]: Upload finished. err: <nil> Aug 7 21:13:22.079763163 (11h33m56.594865742s) [upload=12027]: Starting upload Aug 7 21:13:22.079827372 (11h33m56.594929946s) [upload=12027]: Done starting upload Aug 7 21:13:22.079907596 (11h33m56.595010178s) [upload=12027]: Uploading segments... Aug 7 21:13:22.079966866 (11h33m56.595069445s) [upload=12027]: Got next segment. Inline: true Aug 7 21:13:22.079973866 (11h33m56.595076443s) [upload=12027]: Waiting for error group managing segments... Aug 7 21:13:22.080019901 (11h33m56.595122472s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:22.080024370 (11h33m56.595126947s) [upload=12027]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:22.080029157 (11h33m56.595131732s) [upload=12027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:22.199851686 (11h33m56.714954269s) [upload=12027]: Done uploading segments... Aug 7 21:13:22.199871202 (11h33m56.714973784s) [upload=12027]: Upload finished. err: <nil> Aug 7 21:14:11.619394074 (11h34m46.134496661s) [upload=12040]: Starting upload Aug 7 21:14:11.619503144 (11h34m46.134605739s) [upload=12040]: Done starting upload Aug 7 21:14:11.619622676 (11h34m46.13472524s) [upload=12040]: Uploading segments... Aug 7 21:14:11.619678842 (11h34m46.134781443s) [upload=12040]: Got next segment. Inline: true Aug 7 21:14:11.619698988 (11h34m46.134801579s) [upload=12040]: Waiting for error group managing segments... Aug 7 21:14:11.619745714 (11h34m46.1348483s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:11.619750849 (11h34m46.134853438s) [upload=12040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:11.619754200 (11h34m46.134856779s) [upload=12040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:11.587599715 (11h35m46.102702292s) [upload=12040]: Done uploading segments... Aug 7 21:15:11.587622036 (11h35m46.102724617s) [upload=12040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:11.944029087 (11h35m46.459131673s) [upload=12059]: Starting upload Aug 7 21:15:11.944115611 (11h35m46.459218173s) [upload=12059]: Done starting upload Aug 7 21:15:11.944188828 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000699063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.002119501s FAIL (2025-08-07 21:17:22 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.002145724s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)