Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXX.XX.X..X..] duration: 1.794852834s FAIL (2025-08-07 21:49:06 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.79488181s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 428.553658ms FAIL (2025-08-07 21:49:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 428.572945ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXX.....X...] duration: 787.604968ms FAIL (2025-08-07 21:48:59 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 68549s) [upload=12537]: Done starting upload Aug 7 21:46:22.615396117 (12h6m57.130498679s) [upload=12537]: Uploading segments... Aug 7 21:46:22.615846476 (12h6m57.13094906s) [upload=12537]: Got next segment. Inline: false Aug 7 21:46:22.615968068 (12h6m57.131070645s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:22.615975064 (12h6m57.131077646s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:28.156865165 (12h7m2.671967741s) [seg_pos={0 0}, upload=12537]: Begin upload segment... Aug 7 21:46:28.156917513 (12h7m2.672020096s) [seg_pos={0 0}, upload=12537]: Done begin upload segment. Aug 7 21:46:28.156967463 (12h7m2.672070048s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157000064 (12h7m2.672102649s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157067642 (12h7m2.672170211s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157406065 (12h7m2.672508645s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157549310 (12h7m2.672651889s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157739828 (12h7m2.672842407s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157989895 (12h7m2.673092473s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158099904 (12h7m2.673202487s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158381871 (12h7m2.673484448s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158490401 (12h7m2.673592979s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158794943 (12h7m2.67389751s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158855471 (12h7m2.673958052s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159214922 (12h7m2.674317506s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159509642 (12h7m2.674612208s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161192583 (12h7m2.676295161s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161765903 (12h7m2.676868473s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162066100 (12h7m2.677168688s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162703851 (12h7m2.67780643s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.268596153 (12h7m2.783698719s) [upload=12537]: Next returned nil segment Aug 7 21:46:28.268617977 (12h7m2.783720553s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:46:28.279079135 (12h7m2.794181715s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.290690859 (12h7m2.805793441s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.292185740 (12h7m2.807288316s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.294418015 (12h7m2.80952059s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.295076589 (12h7m2.810179159s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.297491713 (12h7m2.812594291s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.303643978 (12h7m2.818746547s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.304764566 (12h7m2.819867135s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.315655209 (12h7m2.830757788s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.323593569 (12h7m2.838696138s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.324982970 (12h7m2.840085548s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325108866 (12h7m2.840211445s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325989884 (12h7m2.841092452s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361102046 (12h7m2.876204623s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361164635 (12h7m2.876267211s) [seg_pos={0 0}, upload=12537]: Segment reached optimal threshold of 14 pieces. Aug 7 21:46:28.361368094 (12h7m2.876470663s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361578723 (12h7m2.876681288s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eeQieeLS35wwm6vk8rwcTDPPQgYD9vVhVw8WRt84bduXbaW2j): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361874485 (12h7m2.876977051s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o): protocol: read tcp 159.223.143.39:41094->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361908521 (12h7m2.877011091s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d): protocol: read tcp 159.223.143.39:38696->104.36.82.38:4693: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361930237 (12h7m2.877032803s) [seg_pos={0 0}, upload=12537]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:46:28.362049899 (12h7m2.877152488s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:28.362055767 (12h7m2.877158344s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:22.628005735 (12h7m57.143108314s) [upload=12537]: Done uploading segments... Aug 7 21:47:22.628037085 (12h7m57.143139659s) [upload=12537]: Upload finished. err: metaclient: context deadline exceeded Duration: 787.636241ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [..........................XXX........XX..XX.................] duration: 99.241561ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:48:58 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 99.27085ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.X.] duration: 2.310647614s FAIL (2025-08-07 21:48:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.310678608s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.........XX...................] duration: 28.414789ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:48:50 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 28.439109ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 39 out of 60 is failed (threshold=3) [...............X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...XXX.] duration: 230.921256ms FAIL (2025-08-07 21:48:48 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:38.822741159 (12h4m13.337843727s) [upload=12496]: Starting upload Aug 7 21:43:38.822791612 (12h4m13.337894198s) [upload=12496]: Done starting upload Aug 7 21:43:38.822841316 (12h4m13.337943884s) [upload=12496]: Uploading segments... Aug 7 21:43:38.822878518 (12h4m13.337981103s) [upload=12496]: Got next segment. Inline: true Aug 7 21:43:38.822883293 (12h4m13.337985869s) [upload=12496]: Waiting for error group managing segments... Aug 7 21:43:38.822922170 (12h4m13.338024807s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:38.822926518 (12h4m13.338029093s) [upload=12496]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:38.822929823 (12h4m13.338032399s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:38.880695407 (12h4m13.395797984s) [upload=12496]: Done uploading segments... Aug 7 21:43:38.880716578 (12h4m13.395819144s) [upload=12496]: Upload finished. err: <nil> Aug 7 21:44:03.077928978 (12h4m37.593031579s) [upload=12501]: Starting upload Aug 7 21:44:03.078032002 (12h4m37.593134589s) [upload=12501]: Done starting upload Aug 7 21:44:03.078182274 (12h4m37.593284842s) [upload=12501]: Uploading segments... Aug 7 21:44:03.078254104 (12h4m37.59335669s) [upload=12501]: Got next segment. Inline: true Aug 7 21:44:03.078260761 (12h4m37.593363344s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:44:03.078324988 (12h4m37.593427564s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:03.078329732 (12h4m37.593432308s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:03.078333203 (12h4m37.593435779s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:03.150588344 (12h4m37.665690921s) [upload=12501]: Done uploading segments... Aug 7 21:44:03.150609592 (12h4m37.665712177s) [upload=12501]: Upload finished. err: <nil> Aug 7 21:44:56.698719656 (12h5m31.213822219s) [upload=12513]: Starting upload Aug 7 21:44:56.698786775 (12h5m31.213889351s) [upload=12513]: Done starting upload Aug 7 21:44:56.698856755 (12h5m31.213959317s) [upload=12513]: Uploading segments... Aug 7 21:44:56.698908173 (12h5m31.214010758s) [upload=12513]: Got next segment. Inline: true Aug 7 21:44:56.698912338 (12h5m31.214014914s) [upload=12513]: Waiting for error group managing segments... Aug 7 21:44:56.698948274 (12h5m31.21405085s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:56.698965173 (12h5m31.214067748s) [upload=12513]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:56.698967987 (12h5m31.214070563s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:56.781273042 (12h5m31.296375616s) [upload=12513]: Done uploading segments... Aug 7 21:44:56.781296311 (12h5m31.296398893s) [upload=12513]: Upload finished. err: <nil> Aug 7 21:46:10.925701802 (12h6m45.44080437s) [upload=12532]: Starting upload Aug 7 21:46:10.925757954 (12h6m45.440860524s) [upload=12532]: Done starting upload Aug 7 21:46:10.925819534 (12h6m45.440922115s) [upload=12532]: Uploading segments... Aug 7 21:46:10.925869829 (12h6m45.44097242s) [upload=12532]: Got next segment. Inline: true Aug 7 21:46:10.925876475 (12h6m45.440979053s) [upload=12532]: Waiting for error group managing segments... Aug 7 21:46:10.925925157 (12h6m45.441027735s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:10.925930047 (12h6m45.441032624s) [upload=12532]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:10.925933179 (12h6m45.441035756s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:40.893959940 (12h7m15.409062507s) [upload=12532]: Done uploading segments... Aug 7 21:46:40.894015791 (12h7m15.409118368s) [upload=12532]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:17.091619874 (12h7m51.606722441s) [upload=12546]: Starting upload Aug 7 21:47:17.091714186 (12h7m51.606816768s) [upload=12546]: Done starting upload Aug 7 21:47:17.091785367 (12h7m51.606887938s) [upload=12546]: Uploading segments... Aug 7 21:47:17.091863768 (12h7m51.606966354s) [upload=12546]: Got next segment. Inline: true Aug 7 21:47:17.091870088 (12h7m51.606972665s) [upload=12546]: Waiting for error group managing segments... Aug 7 21:47:17.091926672 (12h7m51.607029237s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:17.091932153 (12h7m51.607034729s) [upload=12546]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:17.091935528 (12h7m51.607038105s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:47.062137982 (12h8m21.577240549s) [upload=12546]: Done uploading segments... Aug 7 21:47:47.062157139 (12h8m21.577259705s) [upload=12546]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:55.978154740 (12h8m30.493257315s) [upload=12560]: Starting upload Aug 7 21:47:55.978228943 (12h8m30.493331529s) [upload=12560]: Done starting upload Aug 7 21:47:55.978311008 (12h8m30.493413589s) [upload=12560]: Uploading segments... Aug 7 21:47:55.978374285 (12h8m30.493476868s) [upload=12560]: Got next segment. Inline: true Aug 7 21:47:55.978378891 (12h8m30.493481468s) [upload=12560]: Waiting for error group managing segments... Aug 7 21:47:55.978431682 (12h8m30.493534258s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:55.978436762 (12h8m30.493539338s) [upload=12560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:55.978439848 (12h8m30.493542423s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:25.945081513 (12h9m0.460184093s) [upload=12560]: Done uploading segments... Aug 7 21:48:25.945103520 (12h9m0.460206103s) [upload=12560]: Upload finished. err: metaclient: context deadline exceeded Duration: 230.960074ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 30 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....] duration: 22.047052539s FAIL (2025-08-07 21:48:45 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:02.683404470 (12h4m37.198507059s) [upload=12500]: Starting upload Aug 7 21:44:02.683498983 (12h4m37.198601564s) [upload=12500]: Done starting upload Aug 7 21:44:02.683587655 (12h4m37.198690226s) [upload=12500]: Uploading segments... Aug 7 21:44:02.683635288 (12h4m37.198737882s) [upload=12500]: Got next segment. Inline: true Aug 7 21:44:02.683641097 (12h4m37.198743679s) [upload=12500]: Waiting for error group managing segments... Aug 7 21:44:02.683702304 (12h4m37.198804891s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:02.683709172 (12h4m37.198811757s) [upload=12500]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:02.683713383 (12h4m37.198815961s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.749796670 (12h4m37.264899247s) [upload=12500]: Done uploading segments... Aug 7 21:44:02.749821074 (12h4m37.264923648s) [upload=12500]: Upload finished. err: <nil> Aug 7 21:44:59.056137042 (12h5m33.571239616s) [upload=12515]: Starting upload Aug 7 21:44:59.056204060 (12h5m33.571306636s) [upload=12515]: Done starting upload Aug 7 21:44:59.056275553 (12h5m33.571378122s) [upload=12515]: Uploading segments... Aug 7 21:44:59.056357038 (12h5m33.571459625s) [upload=12515]: Got next segment. Inline: true Aug 7 21:44:59.056364363 (12h5m33.57146694s) [upload=12515]: Waiting for error group managing segments... Aug 7 21:44:59.056439881 (12h5m33.571542451s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:59.056447251 (12h5m33.571549833s) [upload=12515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:59.056453265 (12h5m33.571555847s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:59.018996840 (12h6m33.534099417s) [upload=12515]: Done uploading segments... Aug 7 21:45:59.019015646 (12h6m33.534118222s) [upload=12515]: Upload finished. err: metaclient: context deadline exceeded Duration: 22.047078603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..X..X] duration: 30.000758328s FAIL (2025-08-07 21:48:41 UTC) operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 30.000819665s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...X..] duration: 922.652963ms FAIL (2025-08-07 21:48:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 922.674533ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 691.810713ms FAIL (2025-08-07 21:48:31 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 691.840057ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.X] duration: 30.001380189s FAIL (2025-08-07 21:48:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001402966s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 41 out of 60 is failed (threshold=3) [...........X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....XX..] duration: 356.938187ms FAIL (2025-08-07 21:48:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 356.959852ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 39 out of 60 is failed (threshold=3) [................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...XXX] duration: 30.000551585s FAIL (2025-08-07 21:48:25 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:38.822741159 (12h4m13.337843727s) [upload=12496]: Starting upload Aug 7 21:43:38.822791612 (12h4m13.337894198s) [upload=12496]: Done starting upload Aug 7 21:43:38.822841316 (12h4m13.337943884s) [upload=12496]: Uploading segments... Aug 7 21:43:38.822878518 (12h4m13.337981103s) [upload=12496]: Got next segment. Inline: true Aug 7 21:43:38.822883293 (12h4m13.337985869s) [upload=12496]: Waiting for error group managing segments... Aug 7 21:43:38.822922170 (12h4m13.338024807s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:38.822926518 (12h4m13.338029093s) [upload=12496]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:38.822929823 (12h4m13.338032399s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:38.880695407 (12h4m13.395797984s) [upload=12496]: Done uploading segments... Aug 7 21:43:38.880716578 (12h4m13.395819144s) [upload=12496]: Upload finished. err: <nil> Aug 7 21:44:03.077928978 (12h4m37.593031579s) [upload=12501]: Starting upload Aug 7 21:44:03.078032002 (12h4m37.593134589s) [upload=12501]: Done starting upload Aug 7 21:44:03.078182274 (12h4m37.593284842s) [upload=12501]: Uploading segments... Aug 7 21:44:03.078254104 (12h4m37.59335669s) [upload=12501]: Got next segment. Inline: true Aug 7 21:44:03.078260761 (12h4m37.593363344s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:44:03.078324988 (12h4m37.593427564s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:03.078329732 (12h4m37.593432308s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:03.078333203 (12h4m37.593435779s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:03.150588344 (12h4m37.665690921s) [upload=12501]: Done uploading segments... Aug 7 21:44:03.150609592 (12h4m37.665712177s) [upload=12501]: Upload finished. err: <nil> Aug 7 21:44:56.698719656 (12h5m31.213822219s) [upload=12513]: Starting upload Aug 7 21:44:56.698786775 (12h5m31.213889351s) [upload=12513]: Done starting upload Aug 7 21:44:56.698856755 (12h5m31.213959317s) [upload=12513]: Uploading segments... Aug 7 21:44:56.698908173 (12h5m31.214010758s) [upload=12513]: Got next segment. Inline: true Aug 7 21:44:56.698912338 (12h5m31.214014914s) [upload=12513]: Waiting for error group managing segments... Aug 7 21:44:56.698948274 (12h5m31.21405085s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:56.698965173 (12h5m31.214067748s) [upload=12513]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:56.698967987 (12h5m31.214070563s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:56.781273042 (12h5m31.296375616s) [upload=12513]: Done uploading segments... Aug 7 21:44:56.781296311 (12h5m31.296398893s) [upload=12513]: Upload finished. err: <nil> Aug 7 21:46:10.925701802 (12h6m45.44080437s) [upload=12532]: Starting upload Aug 7 21:46:10.925757954 (12h6m45.440860524s) [upload=12532]: Done starting upload Aug 7 21:46:10.925819534 (12h6m45.440922115s) [upload=12532]: Uploading segments... Aug 7 21:46:10.925869829 (12h6m45.44097242s) [upload=12532]: Got next segment. Inline: true Aug 7 21:46:10.925876475 (12h6m45.440979053s) [upload=12532]: Waiting for error group managing segments... Aug 7 21:46:10.925925157 (12h6m45.441027735s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:10.925930047 (12h6m45.441032624s) [upload=12532]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:10.925933179 (12h6m45.441035756s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:40.893959940 (12h7m15.409062507s) [upload=12532]: Done uploading segments... Aug 7 21:46:40.894015791 (12h7m15.409118368s) [upload=12532]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:17.091619874 (12h7m51.606722441s) [upload=12546]: Starting upload Aug 7 21:47:17.091714186 (12h7m51.606816768s) [upload=12546]: Done starting upload Aug 7 21:47:17.091785367 (12h7m51.606887938s) [upload=12546]: Uploading segments... Aug 7 21:47:17.091863768 (12h7m51.606966354s) [upload=12546]: Got next segment. Inline: true Aug 7 21:47:17.091870088 (12h7m51.606972665s) [upload=12546]: Waiting for error group managing segments... Aug 7 21:47:17.091926672 (12h7m51.607029237s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:17.091932153 (12h7m51.607034729s) [upload=12546]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:17.091935528 (12h7m51.607038105s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:47.062137982 (12h8m21.577240549s) [upload=12546]: Done uploading segments... Aug 7 21:47:47.062157139 (12h8m21.577259705s) [upload=12546]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:55.978154740 (12h8m30.493257315s) [upload=12560]: Starting upload Aug 7 21:47:55.978228943 (12h8m30.493331529s) [upload=12560]: Done starting upload Aug 7 21:47:55.978311008 (12h8m30.493413589s) [upload=12560]: Uploading segments... Aug 7 21:47:55.978374285 (12h8m30.493476868s) [upload=12560]: Got next segment. Inline: true Aug 7 21:47:55.978378891 (12h8m30.493481468s) [upload=12560]: Waiting for error group managing segments... Aug 7 21:47:55.978431682 (12h8m30.493534258s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:55.978436762 (12h8m30.493539338s) [upload=12560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:55.978439848 (12h8m30.493542423s) [upload=12560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:25.945081513 (12h9m0.460184093s) [upload=12560]: Done uploading segments... Aug 7 21:48:25.945103520 (12h9m0.460206103s) [upload=12560]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000577891s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.........XX..................] duration: 43.775857ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:48:17 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 43.812275ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXX.....X..] duration: 1.67644723s FAIL (2025-08-07 21:48:09 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 68549s) [upload=12537]: Done starting upload Aug 7 21:46:22.615396117 (12h6m57.130498679s) [upload=12537]: Uploading segments... Aug 7 21:46:22.615846476 (12h6m57.13094906s) [upload=12537]: Got next segment. Inline: false Aug 7 21:46:22.615968068 (12h6m57.131070645s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:22.615975064 (12h6m57.131077646s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:28.156865165 (12h7m2.671967741s) [seg_pos={0 0}, upload=12537]: Begin upload segment... Aug 7 21:46:28.156917513 (12h7m2.672020096s) [seg_pos={0 0}, upload=12537]: Done begin upload segment. Aug 7 21:46:28.156967463 (12h7m2.672070048s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157000064 (12h7m2.672102649s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157067642 (12h7m2.672170211s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157406065 (12h7m2.672508645s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157549310 (12h7m2.672651889s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157739828 (12h7m2.672842407s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157989895 (12h7m2.673092473s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158099904 (12h7m2.673202487s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158381871 (12h7m2.673484448s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158490401 (12h7m2.673592979s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158794943 (12h7m2.67389751s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158855471 (12h7m2.673958052s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159214922 (12h7m2.674317506s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159509642 (12h7m2.674612208s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161192583 (12h7m2.676295161s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161765903 (12h7m2.676868473s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162066100 (12h7m2.677168688s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162703851 (12h7m2.67780643s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.268596153 (12h7m2.783698719s) [upload=12537]: Next returned nil segment Aug 7 21:46:28.268617977 (12h7m2.783720553s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:46:28.279079135 (12h7m2.794181715s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.290690859 (12h7m2.805793441s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.292185740 (12h7m2.807288316s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.294418015 (12h7m2.80952059s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.295076589 (12h7m2.810179159s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.297491713 (12h7m2.812594291s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.303643978 (12h7m2.818746547s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.304764566 (12h7m2.819867135s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.315655209 (12h7m2.830757788s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.323593569 (12h7m2.838696138s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.324982970 (12h7m2.840085548s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325108866 (12h7m2.840211445s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325989884 (12h7m2.841092452s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361102046 (12h7m2.876204623s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361164635 (12h7m2.876267211s) [seg_pos={0 0}, upload=12537]: Segment reached optimal threshold of 14 pieces. Aug 7 21:46:28.361368094 (12h7m2.876470663s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361578723 (12h7m2.876681288s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eeQieeLS35wwm6vk8rwcTDPPQgYD9vVhVw8WRt84bduXbaW2j): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361874485 (12h7m2.876977051s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o): protocol: read tcp 159.223.143.39:41094->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361908521 (12h7m2.877011091s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d): protocol: read tcp 159.223.143.39:38696->104.36.82.38:4693: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361930237 (12h7m2.877032803s) [seg_pos={0 0}, upload=12537]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:46:28.362049899 (12h7m2.877152488s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:28.362055767 (12h7m2.877158344s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:22.628005735 (12h7m57.143108314s) [upload=12537]: Done uploading segments... Aug 7 21:47:22.628037085 (12h7m57.143139659s) [upload=12537]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.676474733s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [...........................XXX........XX..XX................] duration: 73.98188ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:48:02 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 74.009166ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXX.XX.X..X.] duration: 1.835042619s FAIL (2025-08-07 21:48:00 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.835067047s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 338.890998ms FAIL (2025-08-07 21:47:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 338.92028ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXX.XX.X..X] duration: 30.00113609s FAIL (2025-08-07 21:47:58 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001154837s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 774.462665ms FAIL (2025-08-07 21:47:57 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 774.491155ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 41 out of 60 is failed (threshold=3) [............X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....XX.] duration: 571.991633ms FAIL (2025-08-07 21:47:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 572.003243ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 41 out of 60 is failed (threshold=3) [.............X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....XX] duration: 30.000955049s FAIL (2025-08-07 21:47:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000966091s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [.................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...XX] duration: 30.001270736s FAIL (2025-08-07 21:47:47 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:38.822741159 (12h4m13.337843727s) [upload=12496]: Starting upload Aug 7 21:43:38.822791612 (12h4m13.337894198s) [upload=12496]: Done starting upload Aug 7 21:43:38.822841316 (12h4m13.337943884s) [upload=12496]: Uploading segments... Aug 7 21:43:38.822878518 (12h4m13.337981103s) [upload=12496]: Got next segment. Inline: true Aug 7 21:43:38.822883293 (12h4m13.337985869s) [upload=12496]: Waiting for error group managing segments... Aug 7 21:43:38.822922170 (12h4m13.338024807s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:38.822926518 (12h4m13.338029093s) [upload=12496]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:38.822929823 (12h4m13.338032399s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:38.880695407 (12h4m13.395797984s) [upload=12496]: Done uploading segments... Aug 7 21:43:38.880716578 (12h4m13.395819144s) [upload=12496]: Upload finished. err: <nil> Aug 7 21:44:03.077928978 (12h4m37.593031579s) [upload=12501]: Starting upload Aug 7 21:44:03.078032002 (12h4m37.593134589s) [upload=12501]: Done starting upload Aug 7 21:44:03.078182274 (12h4m37.593284842s) [upload=12501]: Uploading segments... Aug 7 21:44:03.078254104 (12h4m37.59335669s) [upload=12501]: Got next segment. Inline: true Aug 7 21:44:03.078260761 (12h4m37.593363344s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:44:03.078324988 (12h4m37.593427564s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:03.078329732 (12h4m37.593432308s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:03.078333203 (12h4m37.593435779s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:03.150588344 (12h4m37.665690921s) [upload=12501]: Done uploading segments... Aug 7 21:44:03.150609592 (12h4m37.665712177s) [upload=12501]: Upload finished. err: <nil> Aug 7 21:44:56.698719656 (12h5m31.213822219s) [upload=12513]: Starting upload Aug 7 21:44:56.698786775 (12h5m31.213889351s) [upload=12513]: Done starting upload Aug 7 21:44:56.698856755 (12h5m31.213959317s) [upload=12513]: Uploading segments... Aug 7 21:44:56.698908173 (12h5m31.214010758s) [upload=12513]: Got next segment. Inline: true Aug 7 21:44:56.698912338 (12h5m31.214014914s) [upload=12513]: Waiting for error group managing segments... Aug 7 21:44:56.698948274 (12h5m31.21405085s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:56.698965173 (12h5m31.214067748s) [upload=12513]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:56.698967987 (12h5m31.214070563s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:56.781273042 (12h5m31.296375616s) [upload=12513]: Done uploading segments... Aug 7 21:44:56.781296311 (12h5m31.296398893s) [upload=12513]: Upload finished. err: <nil> Aug 7 21:46:10.925701802 (12h6m45.44080437s) [upload=12532]: Starting upload Aug 7 21:46:10.925757954 (12h6m45.440860524s) [upload=12532]: Done starting upload Aug 7 21:46:10.925819534 (12h6m45.440922115s) [upload=12532]: Uploading segments... Aug 7 21:46:10.925869829 (12h6m45.44097242s) [upload=12532]: Got next segment. Inline: true Aug 7 21:46:10.925876475 (12h6m45.440979053s) [upload=12532]: Waiting for error group managing segments... Aug 7 21:46:10.925925157 (12h6m45.441027735s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:10.925930047 (12h6m45.441032624s) [upload=12532]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:10.925933179 (12h6m45.441035756s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:40.893959940 (12h7m15.409062507s) [upload=12532]: Done uploading segments... Aug 7 21:46:40.894015791 (12h7m15.409118368s) [upload=12532]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:47:17.091619874 (12h7m51.606722441s) [upload=12546]: Starting upload Aug 7 21:47:17.091714186 (12h7m51.606816768s) [upload=12546]: Done starting upload Aug 7 21:47:17.091785367 (12h7m51.606887938s) [upload=12546]: Uploading segments... Aug 7 21:47:17.091863768 (12h7m51.606966354s) [upload=12546]: Got next segment. Inline: true Aug 7 21:47:17.091870088 (12h7m51.606972665s) [upload=12546]: Waiting for error group managing segments... Aug 7 21:47:17.091926672 (12h7m51.607029237s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:47:17.091932153 (12h7m51.607034729s) [upload=12546]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:47:17.091935528 (12h7m51.607038105s) [upload=12546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:47.062137982 (12h8m21.577240549s) [upload=12546]: Done uploading segments... Aug 7 21:47:47.062157139 (12h8m21.577259705s) [upload=12546]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001290149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 30 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...] duration: 250.673722ms FAIL (2025-08-07 21:47:47 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:02.683404470 (12h4m37.198507059s) [upload=12500]: Starting upload Aug 7 21:44:02.683498983 (12h4m37.198601564s) [upload=12500]: Done starting upload Aug 7 21:44:02.683587655 (12h4m37.198690226s) [upload=12500]: Uploading segments... Aug 7 21:44:02.683635288 (12h4m37.198737882s) [upload=12500]: Got next segment. Inline: true Aug 7 21:44:02.683641097 (12h4m37.198743679s) [upload=12500]: Waiting for error group managing segments... Aug 7 21:44:02.683702304 (12h4m37.198804891s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:02.683709172 (12h4m37.198811757s) [upload=12500]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:02.683713383 (12h4m37.198815961s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.749796670 (12h4m37.264899247s) [upload=12500]: Done uploading segments... Aug 7 21:44:02.749821074 (12h4m37.264923648s) [upload=12500]: Upload finished. err: <nil> Aug 7 21:44:59.056137042 (12h5m33.571239616s) [upload=12515]: Starting upload Aug 7 21:44:59.056204060 (12h5m33.571306636s) [upload=12515]: Done starting upload Aug 7 21:44:59.056275553 (12h5m33.571378122s) [upload=12515]: Uploading segments... Aug 7 21:44:59.056357038 (12h5m33.571459625s) [upload=12515]: Got next segment. Inline: true Aug 7 21:44:59.056364363 (12h5m33.57146694s) [upload=12515]: Waiting for error group managing segments... Aug 7 21:44:59.056439881 (12h5m33.571542451s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:59.056447251 (12h5m33.571549833s) [upload=12515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:59.056453265 (12h5m33.571555847s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:59.018996840 (12h6m33.534099417s) [upload=12515]: Done uploading segments... Aug 7 21:45:59.019015646 (12h6m33.534118222s) [upload=12515]: Upload finished. err: metaclient: context deadline exceeded Duration: 250.710594ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 37 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..X..] duration: 260.073876ms FAIL (2025-08-07 21:47:35 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 260.104087ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 430.821353ms FAIL (2025-08-07 21:47:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 430.870821ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...X.] duration: 1.046392537s FAIL (2025-08-07 21:47:28 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.046435801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...X] duration: 1m0.001400387s FAIL (2025-08-07 21:47:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001427423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXX.....X.] duration: 1.350620318s FAIL (2025-08-07 21:47:24 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 68549s) [upload=12537]: Done starting upload Aug 7 21:46:22.615396117 (12h6m57.130498679s) [upload=12537]: Uploading segments... Aug 7 21:46:22.615846476 (12h6m57.13094906s) [upload=12537]: Got next segment. Inline: false Aug 7 21:46:22.615968068 (12h6m57.131070645s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:22.615975064 (12h6m57.131077646s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:28.156865165 (12h7m2.671967741s) [seg_pos={0 0}, upload=12537]: Begin upload segment... Aug 7 21:46:28.156917513 (12h7m2.672020096s) [seg_pos={0 0}, upload=12537]: Done begin upload segment. Aug 7 21:46:28.156967463 (12h7m2.672070048s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157000064 (12h7m2.672102649s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157067642 (12h7m2.672170211s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157406065 (12h7m2.672508645s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157549310 (12h7m2.672651889s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157739828 (12h7m2.672842407s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157989895 (12h7m2.673092473s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158099904 (12h7m2.673202487s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158381871 (12h7m2.673484448s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158490401 (12h7m2.673592979s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158794943 (12h7m2.67389751s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158855471 (12h7m2.673958052s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159214922 (12h7m2.674317506s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159509642 (12h7m2.674612208s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161192583 (12h7m2.676295161s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161765903 (12h7m2.676868473s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162066100 (12h7m2.677168688s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162703851 (12h7m2.67780643s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.268596153 (12h7m2.783698719s) [upload=12537]: Next returned nil segment Aug 7 21:46:28.268617977 (12h7m2.783720553s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:46:28.279079135 (12h7m2.794181715s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.290690859 (12h7m2.805793441s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.292185740 (12h7m2.807288316s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.294418015 (12h7m2.80952059s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.295076589 (12h7m2.810179159s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.297491713 (12h7m2.812594291s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.303643978 (12h7m2.818746547s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.304764566 (12h7m2.819867135s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.315655209 (12h7m2.830757788s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.323593569 (12h7m2.838696138s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.324982970 (12h7m2.840085548s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325108866 (12h7m2.840211445s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325989884 (12h7m2.841092452s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361102046 (12h7m2.876204623s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361164635 (12h7m2.876267211s) [seg_pos={0 0}, upload=12537]: Segment reached optimal threshold of 14 pieces. Aug 7 21:46:28.361368094 (12h7m2.876470663s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361578723 (12h7m2.876681288s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eeQieeLS35wwm6vk8rwcTDPPQgYD9vVhVw8WRt84bduXbaW2j): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361874485 (12h7m2.876977051s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o): protocol: read tcp 159.223.143.39:41094->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361908521 (12h7m2.877011091s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d): protocol: read tcp 159.223.143.39:38696->104.36.82.38:4693: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361930237 (12h7m2.877032803s) [seg_pos={0 0}, upload=12537]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:46:28.362049899 (12h7m2.877152488s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:28.362055767 (12h7m2.877158344s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:22.628005735 (12h7m57.143108314s) [upload=12537]: Done uploading segments... Aug 7 21:47:22.628037085 (12h7m57.143139659s) [upload=12537]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.350638837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [..............X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....X] duration: 30.001532552s FAIL (2025-08-07 21:47:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001561708s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXX.....X] duration: 1m0.047293051s FAIL (2025-08-07 21:47:22 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 68549s) [upload=12537]: Done starting upload Aug 7 21:46:22.615396117 (12h6m57.130498679s) [upload=12537]: Uploading segments... Aug 7 21:46:22.615846476 (12h6m57.13094906s) [upload=12537]: Got next segment. Inline: false Aug 7 21:46:22.615968068 (12h6m57.131070645s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:22.615975064 (12h6m57.131077646s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:46:28.156865165 (12h7m2.671967741s) [seg_pos={0 0}, upload=12537]: Begin upload segment... Aug 7 21:46:28.156917513 (12h7m2.672020096s) [seg_pos={0 0}, upload=12537]: Done begin upload segment. Aug 7 21:46:28.156967463 (12h7m2.672070048s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157000064 (12h7m2.672102649s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157067642 (12h7m2.672170211s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157406065 (12h7m2.672508645s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157549310 (12h7m2.672651889s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157739828 (12h7m2.672842407s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.157989895 (12h7m2.673092473s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158099904 (12h7m2.673202487s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158381871 (12h7m2.673484448s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158490401 (12h7m2.673592979s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158794943 (12h7m2.67389751s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.158855471 (12h7m2.673958052s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159214922 (12h7m2.674317506s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.159509642 (12h7m2.674612208s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161192583 (12h7m2.676295161s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.161765903 (12h7m2.676868473s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162066100 (12h7m2.677168688s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.162703851 (12h7m2.67780643s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Uploading piece... Aug 7 21:46:28.268596153 (12h7m2.783698719s) [upload=12537]: Next returned nil segment Aug 7 21:46:28.268617977 (12h7m2.783720553s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:46:28.279079135 (12h7m2.794181715s) [noise=true , address=104.36.82.22:4529 , piece_id=XX6WFY3JF6BQSSTWJ5X7O2MFYS5HTGLX2NDCNY4CVWB5OVWJWYFQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.290690859 (12h7m2.805793441s) [noise=true , address=104.36.82.61:4394 , piece_id=5GHVG3NOARD52VKHQ74NAPJIFDQYTHRSWI2U6TGPT64YC4EZVDBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.292185740 (12h7m2.807288316s) [noise=true , address=104.36.82.62:4436 , piece_id=FJVZ36YLHDRZCVUMMZTBVDBTMUVPIHZKGLKKA6UD5GRLOHAWOLBQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.294418015 (12h7m2.80952059s) [noise=true , address=104.36.82.59:4304 , piece_id=W5W6TQZ7JL7EDCYGGTUERVAPIEK376SCHYG5KDXUONQCJCQUW46A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.295076589 (12h7m2.810179159s) [noise=true , address=104.36.82.45:4918 , piece_id=D5JWM3VZEZACAQ2LNGZ2VUHCR3DHWAQI3QB3FXYZUXVGA3GMYOFA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.297491713 (12h7m2.812594291s) [noise=true , address=104.36.82.60:4362 , piece_id=JOAYEHXCDTD62DBPDO2S32UFN2KY56KPOCNT47MWE6SDNUOCK7YA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.303643978 (12h7m2.818746547s) [noise=true , address=45.91.116.14:4202 , piece_id=EK5OTA7WTPTPYYSDD65ZXO4T5MHZWJWCEGWPEFPPWXUBDMMGRRTA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.304764566 (12h7m2.819867135s) [noise=true , address=77.90.31.12:4614 , piece_id=2IBWHRA3XHYFICTJ3Q2JIZBAG2JCZPFNBG2LNQ5ZCHIZ3QTR66HA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.315655209 (12h7m2.830757788s) [noise=true , address=104.36.82.21:4498 , piece_id=MNQZMVJ72H23WME6DFVFKVAGZWZ3WYNQMIISZSIT2JKCRORVLDWA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.323593569 (12h7m2.838696138s) [noise=true , address=104.36.82.16:4321 , piece_id=45WVSW7DMKKOM3QBJLSKPIAVUKUEHJCL4FELENSIMZ7NWP5DETZQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.324982970 (12h7m2.840085548s) [noise=true , address=104.36.82.10:4091 , piece_id=XOIC6TBUSWRTJ25TXOCR3J6XC5BBPZMHRS3KQH4AB6Q6DJXRJB4A, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325108866 (12h7m2.840211445s) [noise=true , address=104.36.82.5:4870 , piece_id=DAFNVQEHQWVCAQUXYZEO4PI3JGYNTVD6AAJP4JNBS6LUXBZKLIEQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.325989884 (12h7m2.841092452s) [noise=true , address=104.36.82.2:4735 , piece_id=V57PWSXMX2KICKCOEWN3PZFPKX33VSZFBCAT7ZPXXFPG56LERAQQ, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361102046 (12h7m2.876204623s) [noise=true , address=79.127.220.97:4395 , piece_id=4CMF2AH4G5I5BDRFLMQLOJFF3457OZOJ2LMPEK643KH2I77PZAIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: <nil> Aug 7 21:46:28.361164635 (12h7m2.876267211s) [seg_pos={0 0}, upload=12537]: Segment reached optimal threshold of 14 pieces. Aug 7 21:46:28.361368094 (12h7m2.876470663s) [noise=true , address=104.36.82.54:4160 , piece_id=3MRQI434K7PJKJ2EPH4HKND4TGI3IBJ3JRNVCC6V36FNN4UWDTBA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361578723 (12h7m2.876681288s) [noise=true , address=79.127.201.215:4348 , piece_id=SXQQKH4WBTYHX6WJFW62T2F3TAEVLPSAS2TMWVRSJVQTHSKMHQEA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eeQieeLS35wwm6vk8rwcTDPPQgYD9vVhVw8WRt84bduXbaW2j): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:28.361874485 (12h7m2.876977051s) [noise=true , address=104.36.82.9:4015 , piece_id=RRFBF5BFR5NPJ2RCQ3XGAMXWYTQ4UOB3XE5FXFY3XSR4274DQNQA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o): protocol: read tcp 159.223.143.39:41094->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361908521 (12h7m2.877011091s) [noise=true , address=104.36.82.38:4693 , piece_id=VTETVVCYOHZBA4PHI6RSDBILQAXTR2BOCEC72UWXDCYN4YK2IDIA, seg_pos={0 0}, upload=12537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d): protocol: read tcp 159.223.143.39:38696->104.36.82.38:4693: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:28.361930237 (12h7m2.877032803s) [seg_pos={0 0}, upload=12537]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:46:28.362049899 (12h7m2.877152488s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:28.362055767 (12h7m2.877158344s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:22.628005735 (12h7m57.143108314s) [upload=12537]: Done uploading segments... Aug 7 21:47:22.628037085 (12h7m57.143139659s) [upload=12537]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.047320472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 30 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..] duration: 394.310215ms FAIL (2025-08-07 21:47:20 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:02.683404470 (12h4m37.198507059s) [upload=12500]: Starting upload Aug 7 21:44:02.683498983 (12h4m37.198601564s) [upload=12500]: Done starting upload Aug 7 21:44:02.683587655 (12h4m37.198690226s) [upload=12500]: Uploading segments... Aug 7 21:44:02.683635288 (12h4m37.198737882s) [upload=12500]: Got next segment. Inline: true Aug 7 21:44:02.683641097 (12h4m37.198743679s) [upload=12500]: Waiting for error group managing segments... Aug 7 21:44:02.683702304 (12h4m37.198804891s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:02.683709172 (12h4m37.198811757s) [upload=12500]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:02.683713383 (12h4m37.198815961s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.749796670 (12h4m37.264899247s) [upload=12500]: Done uploading segments... Aug 7 21:44:02.749821074 (12h4m37.264923648s) [upload=12500]: Upload finished. err: <nil> Aug 7 21:44:59.056137042 (12h5m33.571239616s) [upload=12515]: Starting upload Aug 7 21:44:59.056204060 (12h5m33.571306636s) [upload=12515]: Done starting upload Aug 7 21:44:59.056275553 (12h5m33.571378122s) [upload=12515]: Uploading segments... Aug 7 21:44:59.056357038 (12h5m33.571459625s) [upload=12515]: Got next segment. Inline: true Aug 7 21:44:59.056364363 (12h5m33.57146694s) [upload=12515]: Waiting for error group managing segments... Aug 7 21:44:59.056439881 (12h5m33.571542451s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:59.056447251 (12h5m33.571549833s) [upload=12515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:59.056453265 (12h5m33.571555847s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:59.018996840 (12h6m33.534099417s) [upload=12515]: Done uploading segments... Aug 7 21:45:59.019015646 (12h6m33.534118222s) [upload=12515]: Upload finished. err: metaclient: context deadline exceeded Duration: 394.336812ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.........XX.................] duration: 64.03154ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:47:15 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 64.057175ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X.] duration: 496.319167ms FAIL (2025-08-07 21:47:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.366907ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.615483331s FAIL (2025-08-07 21:47:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 2.615516361s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [............................XXX........XX..XX...............] duration: 81.278913ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:47:07 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 81.31271ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 795.015876ms FAIL (2025-08-07 21:47:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 795.046939ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....X] duration: 30.001609113s FAIL (2025-08-07 21:46:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001633301s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 37 out of 60 is failed (threshold=3) [..................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...X] duration: 30.001391498s FAIL (2025-08-07 21:46:40 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:38.822741159 (12h4m13.337843727s) [upload=12496]: Starting upload Aug 7 21:43:38.822791612 (12h4m13.337894198s) [upload=12496]: Done starting upload Aug 7 21:43:38.822841316 (12h4m13.337943884s) [upload=12496]: Uploading segments... Aug 7 21:43:38.822878518 (12h4m13.337981103s) [upload=12496]: Got next segment. Inline: true Aug 7 21:43:38.822883293 (12h4m13.337985869s) [upload=12496]: Waiting for error group managing segments... Aug 7 21:43:38.822922170 (12h4m13.338024807s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:38.822926518 (12h4m13.338029093s) [upload=12496]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:38.822929823 (12h4m13.338032399s) [upload=12496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:38.880695407 (12h4m13.395797984s) [upload=12496]: Done uploading segments... Aug 7 21:43:38.880716578 (12h4m13.395819144s) [upload=12496]: Upload finished. err: <nil> Aug 7 21:44:03.077928978 (12h4m37.593031579s) [upload=12501]: Starting upload Aug 7 21:44:03.078032002 (12h4m37.593134589s) [upload=12501]: Done starting upload Aug 7 21:44:03.078182274 (12h4m37.593284842s) [upload=12501]: Uploading segments... Aug 7 21:44:03.078254104 (12h4m37.59335669s) [upload=12501]: Got next segment. Inline: true Aug 7 21:44:03.078260761 (12h4m37.593363344s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:44:03.078324988 (12h4m37.593427564s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:03.078329732 (12h4m37.593432308s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:03.078333203 (12h4m37.593435779s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:03.150588344 (12h4m37.665690921s) [upload=12501]: Done uploading segments... Aug 7 21:44:03.150609592 (12h4m37.665712177s) [upload=12501]: Upload finished. err: <nil> Aug 7 21:44:56.698719656 (12h5m31.213822219s) [upload=12513]: Starting upload Aug 7 21:44:56.698786775 (12h5m31.213889351s) [upload=12513]: Done starting upload Aug 7 21:44:56.698856755 (12h5m31.213959317s) [upload=12513]: Uploading segments... Aug 7 21:44:56.698908173 (12h5m31.214010758s) [upload=12513]: Got next segment. Inline: true Aug 7 21:44:56.698912338 (12h5m31.214014914s) [upload=12513]: Waiting for error group managing segments... Aug 7 21:44:56.698948274 (12h5m31.21405085s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:56.698965173 (12h5m31.214067748s) [upload=12513]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:56.698967987 (12h5m31.214070563s) [upload=12513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:56.781273042 (12h5m31.296375616s) [upload=12513]: Done uploading segments... Aug 7 21:44:56.781296311 (12h5m31.296398893s) [upload=12513]: Upload finished. err: <nil> Aug 7 21:46:10.925701802 (12h6m45.44080437s) [upload=12532]: Starting upload Aug 7 21:46:10.925757954 (12h6m45.440860524s) [upload=12532]: Done starting upload Aug 7 21:46:10.925819534 (12h6m45.440922115s) [upload=12532]: Uploading segments... Aug 7 21:46:10.925869829 (12h6m45.44097242s) [upload=12532]: Got next segment. Inline: true Aug 7 21:46:10.925876475 (12h6m45.440979053s) [upload=12532]: Waiting for error group managing segments... Aug 7 21:46:10.925925157 (12h6m45.441027735s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:10.925930047 (12h6m45.441032624s) [upload=12532]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:10.925933179 (12h6m45.441035756s) [upload=12532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:46:40.893959940 (12h7m15.409062507s) [upload=12532]: Done uploading segments... Aug 7 21:46:40.894015791 (12h7m15.409118368s) [upload=12532]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001459315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 37 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..X.] duration: 368.242273ms FAIL (2025-08-07 21:46:35 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 368.263369ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 37 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..X] duration: 30.000510831s FAIL (2025-08-07 21:46:34 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000530022s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [................................X.........XX................] duration: 34.159533ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:46:29 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 34.182171ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.............................XXX........XX..XX..............] duration: 39.638997ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:46:19 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 39.67011ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXX.XX.X..] duration: 1.77752208s FAIL (2025-08-07 21:46:18 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.777550569s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.835518139s FAIL (2025-08-07 21:46:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 1.835543106s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 812.755904ms FAIL (2025-08-07 21:46:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 812.782832ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 30 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 245.21341ms FAIL (2025-08-07 21:46:08 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:02.683404470 (12h4m37.198507059s) [upload=12500]: Starting upload Aug 7 21:44:02.683498983 (12h4m37.198601564s) [upload=12500]: Done starting upload Aug 7 21:44:02.683587655 (12h4m37.198690226s) [upload=12500]: Uploading segments... Aug 7 21:44:02.683635288 (12h4m37.198737882s) [upload=12500]: Got next segment. Inline: true Aug 7 21:44:02.683641097 (12h4m37.198743679s) [upload=12500]: Waiting for error group managing segments... Aug 7 21:44:02.683702304 (12h4m37.198804891s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:02.683709172 (12h4m37.198811757s) [upload=12500]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:02.683713383 (12h4m37.198815961s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.749796670 (12h4m37.264899247s) [upload=12500]: Done uploading segments... Aug 7 21:44:02.749821074 (12h4m37.264923648s) [upload=12500]: Upload finished. err: <nil> Aug 7 21:44:59.056137042 (12h5m33.571239616s) [upload=12515]: Starting upload Aug 7 21:44:59.056204060 (12h5m33.571306636s) [upload=12515]: Done starting upload Aug 7 21:44:59.056275553 (12h5m33.571378122s) [upload=12515]: Uploading segments... Aug 7 21:44:59.056357038 (12h5m33.571459625s) [upload=12515]: Got next segment. Inline: true Aug 7 21:44:59.056364363 (12h5m33.57146694s) [upload=12515]: Waiting for error group managing segments... Aug 7 21:44:59.056439881 (12h5m33.571542451s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:59.056447251 (12h5m33.571549833s) [upload=12515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:59.056453265 (12h5m33.571555847s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:59.018996840 (12h6m33.534099417s) [upload=12515]: Done uploading segments... Aug 7 21:45:59.019015646 (12h6m33.534118222s) [upload=12515]: Upload finished. err: metaclient: context deadline exceeded Duration: 245.239816ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 30 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 1m0.000812763s FAIL (2025-08-07 21:45:59 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:02.683404470 (12h4m37.198507059s) [upload=12500]: Starting upload Aug 7 21:44:02.683498983 (12h4m37.198601564s) [upload=12500]: Done starting upload Aug 7 21:44:02.683587655 (12h4m37.198690226s) [upload=12500]: Uploading segments... Aug 7 21:44:02.683635288 (12h4m37.198737882s) [upload=12500]: Got next segment. Inline: true Aug 7 21:44:02.683641097 (12h4m37.198743679s) [upload=12500]: Waiting for error group managing segments... Aug 7 21:44:02.683702304 (12h4m37.198804891s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:02.683709172 (12h4m37.198811757s) [upload=12500]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:02.683713383 (12h4m37.198815961s) [upload=12500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.749796670 (12h4m37.264899247s) [upload=12500]: Done uploading segments... Aug 7 21:44:02.749821074 (12h4m37.264923648s) [upload=12500]: Upload finished. err: <nil> Aug 7 21:44:59.056137042 (12h5m33.571239616s) [upload=12515]: Starting upload Aug 7 21:44:59.056204060 (12h5m33.571306636s) [upload=12515]: Done starting upload Aug 7 21:44:59.056275553 (12h5m33.571378122s) [upload=12515]: Uploading segments... Aug 7 21:44:59.056357038 (12h5m33.571459625s) [upload=12515]: Got next segment. Inline: true Aug 7 21:44:59.056364363 (12h5m33.57146694s) [upload=12515]: Waiting for error group managing segments... Aug 7 21:44:59.056439881 (12h5m33.571542451s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:59.056447251 (12h5m33.571549833s) [upload=12515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:59.056453265 (12h5m33.571555847s) [upload=12515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:59.018996840 (12h6m33.534099417s) [upload=12515]: Done uploading segments... Aug 7 21:45:59.019015646 (12h6m33.534118222s) [upload=12515]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000833162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 39 out of 60 is failed (threshold=3) [...............X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 277.260685ms FAIL (2025-08-07 21:45:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 277.286194ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)