Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [................................................XXXXXXX.....] duration: 466.899498ms trace: https://jaeger.dev.storj.io/trace/35e31fda89825398 FAIL (2025-11-30 10:06:11 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 467.076261ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.................................................XXXXXXX....] duration: 385.063847ms trace: https://jaeger.dev.storj.io/trace/35e31fda89825398 FAIL (2025-11-30 10:05:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 385.085092ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration: 757.924787ms trace: https://jaeger.dev.storj.io/trace/1075f18a004aadd1 FAIL (2025-11-30 10:05:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 757.946972ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXXXX.....] duration: 384.053682ms trace: https://jaeger.dev.storj.io/trace/55d9d79c11454024 FAIL (2025-11-30 10:05:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments... Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments... Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 384.075981ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration: 340.425685ms trace: https://jaeger.dev.storj.io/trace/271f991f86732ab4 FAIL (2025-11-30 10:05:18 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments... Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment... Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment. Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments... Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments... Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 340.465695ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..................................................XXXXXXX...] duration: 417.304784ms trace: https://jaeger.dev.storj.io/trace/35e31fda89825398 FAIL (2025-11-30 10:05:12 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 417.332553ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 465.001314ms trace: https://jaeger.dev.storj.io/trace/1075f18a004aadd1 FAIL (2025-11-30 10:04:38 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.023586ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration: 896.024241ms trace: https://jaeger.dev.storj.io/trace/55d9d79c11454024 FAIL (2025-11-30 10:04:33 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments... Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments... Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 896.049484ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 349.216057ms trace: https://jaeger.dev.storj.io/trace/271f991f86732ab4 FAIL (2025-11-30 10:04:28 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments... Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment... Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment. Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments... Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments... Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 349.306114ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...................................................XXXXXXX..] duration: 406.475283ms trace: https://jaeger.dev.storj.io/trace/35e31fda89825398 FAIL (2025-11-30 10:04:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 406.499671ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 433.493408ms trace: https://jaeger.dev.storj.io/trace/55d9d79c11454024 FAIL (2025-11-30 10:04:09 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments... Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments... Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 433.515796ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 546.87102ms trace: https://jaeger.dev.storj.io/trace/1075f18a004aadd1 FAIL (2025-11-30 10:04:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 546.902824ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 441.413388ms trace: https://jaeger.dev.storj.io/trace/55d9d79c11454024 FAIL (2025-11-30 10:03:17 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments... Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments... Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 441.455325ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 382.239753ms trace: https://jaeger.dev.storj.io/trace/271f991f86732ab4 FAIL (2025-11-30 10:03:16 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments... Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment... Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment. Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments... Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments... Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 382.267582ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 12.325944081s trace: https://jaeger.dev.storj.io/trace/1075f18a004aadd1 FAIL (2025-11-30 10:03:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.325957828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....................................................XXXXXXX.] duration: 2.441443276s trace: https://jaeger.dev.storj.io/trace/35e31fda89825398 FAIL (2025-11-30 10:03:04 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.441480004s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.000701055s trace: https://jaeger.dev.storj.io/trace/1075f18a004aadd1 FAIL (2025-11-30 10:02:56 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000730964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.026477247s trace: https://jaeger.dev.storj.io/trace/55d9d79c11454024 FAIL (2025-11-30 10:02:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece... Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments... Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil> Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments... Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.026490993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.031938595s trace: https://jaeger.dev.storj.io/trace/271f991f86732ab4 FAIL (2025-11-30 10:02:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments... Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment... Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment. Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece... Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments... Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil> Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments... Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.031976231s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....................................................XXXXXXX] duration: 5.002009653s trace: https://jaeger.dev.storj.io/trace/35e31fda89825398 FAIL (2025-11-30 10:02:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002037741s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.000709534s trace: https://jaeger.dev.storj.io/trace/5c773b9743fa4ae5 FAIL (2025-11-30 10:01:56 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000778779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......................................................XXXXXX] duration: 5.000790342s trace: https://jaeger.dev.storj.io/trace/6d1ed51ee50e0469 FAIL (2025-11-30 10:01:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000804353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.026234726s trace: https://jaeger.dev.storj.io/trace/1390a11e65a9cdb FAIL (2025-11-30 10:01:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 13m49.844114911s) [seg_pos={0 0}, upload=474822]: Done begin upload segment. Nov 30 10:00:25.076593465 (456h13m49.844276901s) [noise=true , address=77.90.31.13:4652 , piece_id=QJ6CIQ73JVYAKI2ILEPX7Y34G3UHZVBJ2UEOQGRAWSG4AWPC5RGA, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.076650337 (456h13m49.844333768s) [noise=true , address=64.49.13.139:4477 , piece_id=47YTFCRAADOQECV552D7UT6OCJ35YJZOB6O4YTPLGRTVAMQSU4CA, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.079704035 (456h13m49.847387465s) [noise=true , address=77.90.31.10:4548 , piece_id=TJF4P662P5HEFXYDXALVRUTXDCR7QVJMZNDXKHEFRKHTFX2PQ74A, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.080844524 (456h13m49.848527961s) [noise=true , address=45.91.116.7:4040 , piece_id=WAXZOET7X3NB557AXZTP6L4FPI5KQF374W5T6JB3QHT4JIL3IJPQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.081662912 (456h13m49.849346349s) [noise=true , address=77.90.31.14:4695 , piece_id=XMEUOZVPXT5AJSOM4NLI2XNA57AJNXP6P6X62TER65LX7EAUJFMQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.082513467 (456h13m49.8501969s) [noise=true , address=45.91.116.9:4271 , piece_id=MMKM2PYCRH5ZP57PCSBHQCVOY4LZYFJCNFHWLNB2VPRBBU4DBWSQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.085039690 (456h13m49.852723104s) [noise=true , address=45.91.116.4:4957 , piece_id=BJ3IBZVVJCPD366TUXYZDHMCLEZIHY5IHOY5MAFGYQMEMHFRBTEQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.087586214 (456h13m49.855269651s) [noise=true , address=77.90.31.21:4905 , piece_id=ZU54FTOJ3OYIAAWDYJR54ZDBEGAXDLTCZYLFI63J4IZZ6R35GPOQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.087833753 (456h13m49.855517191s) [noise=true , address=77.90.31.11:4564 , piece_id=U4SFX4MUSFJM67FE7SE3YJO5ROY75XUWMZKZAKYBEGHIIAZZMAJA, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.089365045 (456h13m49.857048478s) [noise=true , address=64.49.13.169:4705 , piece_id=JP24MSYAUNKQT7NRKV472GJ4RBOR4XO2FEC3L6N2NF4SRRAEXMSQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.090448695 (456h13m49.858132128s) [noise=true , address=64.49.13.140:4516 , piece_id=FM2Y2EJGYNOZPY77XABANL6FSVV7HYHJZLOZZ2BQVYU2PCAWCESQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.090951145 (456h13m49.858634576s) [noise=true , address=77.90.31.18:4823 , piece_id=T5H3ZDC6OCNBO6GREFS2HXWD5YYOYCCKID2YTGBOKOKQMKG7B3TQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.093010878 (456h13m49.860694312s) [noise=true , address=45.91.116.13:4172 , piece_id=YOWPUZD3NSFKXAXYROIS7VI4NWKD3K7T2W42ZOU72VN5UHJDOSGQ, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.093329784 (456h13m49.861013216s) [noise=true , address=45.91.116.8:4051 , piece_id=IGYF7V3MD7BEN63GAIPFVDLS7MDFPBJ5DMBW2AFV77JTDSBH75PA, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.094887347 (456h13m49.862570777s) [noise=true , address=45.91.116.6:4002 , piece_id=65CPF2RW6FPD2YSN3CBOPIOXZTW3C33GJ7V7PK5RVYVKUVZGAR2A, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.095522042 (456h13m49.863205473s) [noise=true , address=77.90.31.15:4700 , piece_id=GGYKBSQ4EC4I25MOY3KEMAVKOUL4L3772QEV2GL6T75OXB6ATTFA, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.098729938 (456h13m49.866413372s) [noise=true , address=45.91.116.10:4097 , piece_id=OPMSDWPTNDND3LFKVDNAD2QEK37JTMF5I6W7ZGYJN54AXPZYDEBA, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.101025278 (456h13m49.868708703s) [noise=true , address=45.91.116.5:4316 , piece_id=BOVP5HG2U527Z2FZNR75UPVJRCN6R27RRABUBUALZ53AJCCYVQ3Q, seg_pos={0 0}, upload=474822]: Uploading piece... Nov 30 10:00:25.126289278 (456h13m49.89397269s) [upload=474822]: Next returned nil segment Nov 30 10:00:25.126302190 (456h13m49.893985616s) [upload=474822]: Waiting for error group managing segments... Nov 30 10:00:25.130361810 (456h13m49.898045238s) [noise=true , address=45.91.116.6:4002 , piece_id=65CPF2RW6FPD2YSN3CBOPIOXZTW3C33GJ7V7PK5RVYVKUVZGAR2A, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.130411430 (456h13m49.898094858s) [noise=true , address=45.91.116.7:4040 , piece_id=WAXZOET7X3NB557AXZTP6L4FPI5KQF374W5T6JB3QHT4JIL3IJPQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.130730205 (456h13m49.898413634s) [noise=true , address=45.91.116.13:4172 , piece_id=YOWPUZD3NSFKXAXYROIS7VI4NWKD3K7T2W42ZOU72VN5UHJDOSGQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.130974358 (456h13m49.898657786s) [noise=true , address=45.91.116.9:4271 , piece_id=MMKM2PYCRH5ZP57PCSBHQCVOY4LZYFJCNFHWLNB2VPRBBU4DBWSQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.131064570 (456h13m49.898747999s) [noise=true , address=45.91.116.4:4957 , piece_id=BJ3IBZVVJCPD366TUXYZDHMCLEZIHY5IHOY5MAFGYQMEMHFRBTEQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.131458379 (456h13m49.899141807s) [noise=true , address=45.91.116.10:4097 , piece_id=OPMSDWPTNDND3LFKVDNAD2QEK37JTMF5I6W7ZGYJN54AXPZYDEBA, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.144757486 (456h13m49.912440911s) [noise=true , address=45.91.116.8:4051 , piece_id=IGYF7V3MD7BEN63GAIPFVDLS7MDFPBJ5DMBW2AFV77JTDSBH75PA, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.149359973 (456h13m49.917043401s) [noise=true , address=45.91.116.5:4316 , piece_id=BOVP5HG2U527Z2FZNR75UPVJRCN6R27RRABUBUALZ53AJCCYVQ3Q, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.689358281 (456h13m50.457041708s) [noise=true , address=64.49.13.169:4705 , piece_id=JP24MSYAUNKQT7NRKV472GJ4RBOR4XO2FEC3L6N2NF4SRRAEXMSQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.710164523 (456h13m50.477847949s) [noise=true , address=64.49.13.139:4477 , piece_id=47YTFCRAADOQECV552D7UT6OCJ35YJZOB6O4YTPLGRTVAMQSU4CA, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:00:25.761437069 (456h13m50.529120494s) [noise=true , address=64.49.13.140:4516 , piece_id=FM2Y2EJGYNOZPY77XABANL6FSVV7HYHJZLOZZ2BQVYU2PCAWCESQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: <nil> Nov 30 10:01:24.999585663 (456h14m49.767269097s) [noise=true , address=77.90.31.21:4905 , piece_id=ZU54FTOJ3OYIAAWDYJR54ZDBEGAXDLTCZYLFI63J4IZZ6R35GPOQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: rpc: dial tcp 77.90.31.21:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:24.999621746 (456h14m49.767305181s) [noise=true , address=77.90.31.10:4548 , piece_id=TJF4P662P5HEFXYDXALVRUTXDCR7QVJMZNDXKHEFRKHTFX2PQ74A, seg_pos={0 0}, upload=474822]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:24.999871154 (456h14m49.767554589s) [noise=true , address=77.90.31.11:4564 , piece_id=U4SFX4MUSFJM67FE7SE3YJO5ROY75XUWMZKZAKYBEGHIIAZZMAJA, seg_pos={0 0}, upload=474822]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.000148458 (456h14m49.767831912s) [noise=true , address=77.90.31.18:4823 , piece_id=T5H3ZDC6OCNBO6GREFS2HXWD5YYOYCCKID2YTGBOKOKQMKG7B3TQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.000427475 (456h14m49.768110924s) [noise=true , address=77.90.31.14:4695 , piece_id=XMEUOZVPXT5AJSOM4NLI2XNA57AJNXP6P6X62TER65LX7EAUJFMQ, seg_pos={0 0}, upload=474822]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 159.223.143.39:36516->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.000476372 (456h14m49.768159826s) [noise=true , address=77.90.31.13:4652 , piece_id=QJ6CIQ73JVYAKI2ILEPX7Y34G3UHZVBJ2UEOQGRAWSG4AWPC5RGA, seg_pos={0 0}, upload=474822]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: rpc: dial tcp 77.90.31.13:4652: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.000603937 (456h14m49.768287388s) [noise=true , address=77.90.31.15:4700 , piece_id=GGYKBSQ4EC4I25MOY3KEMAVKOUL4L3772QEV2GL6T75OXB6ATTFA, seg_pos={0 0}, upload=474822]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 159.223.143.39:58382->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.000629811 (456h14m49.768313243s) [seg_pos={0 0}, upload=474822]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:01:25.023397706 (456h14m49.791081135s) [upload=474822]: Done uploading segments... Nov 30 10:01:25.023452474 (456h14m49.791135901s) [upload=474822]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.026326809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.020915687s trace: https://jaeger.dev.storj.io/trace/16cf90e1dd1c618b FAIL (2025-11-30 10:01:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Nov 30 10:00:25.563665596 (456h13m50.331349026s) [upload=474823]: Starting upload Nov 30 10:00:25.563739992 (456h13m50.331423428s) [upload=474823]: Done starting upload Nov 30 10:00:25.564145037 (456h13m50.331828464s) [upload=474823]: Uploading segments... Nov 30 10:00:25.564207902 (456h13m50.331891328s) [upload=474823]: Got next segment. Inline: false Nov 30 10:00:25.564311455 (456h13m50.331994883s) [upload=474823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 10:00:25.564317554 (456h13m50.332000979s) [upload=474823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 10:00:25.591370722 (456h13m50.359054153s) [seg_pos={0 0}, upload=474823]: Begin upload segment... Nov 30 10:00:25.591477775 (456h13m50.359161211s) [seg_pos={0 0}, upload=474823]: Done begin upload segment. Nov 30 10:00:25.591528327 (456h13m50.359211757s) [noise=true , address=45.91.116.14:4214 , piece_id=5UDQZTYWNVXVSLCYVRD5B7LENDPXEPKIG7AGDHVWUCBN2SVCODCA, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.591546865 (456h13m50.3592303s) [noise=true , address=64.49.13.26:4955 , piece_id=QS27N72UQ2KYYPZPA6IP3433GT5QLCGCK4Z6DPLGISRM427A457A, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.592676871 (456h13m50.360360307s) [noise=true , address=145.79.227.3:4183 , piece_id=KINVB5A5IEEL5RRBQI62WVL3MCYLV5RLWGI4RSKZTMB76RRD6YVQ, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.593120266 (456h13m50.360803696s) [noise=true , address=145.79.227.7:4311 , piece_id=ZNNFEWO24LQSQDGHQPDPPFJOU3JNXRV7PPNQQHVQWKEO4CNCMJXA, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.593461151 (456h13m50.361144584s) [noise=true , address=45.91.116.6:4005 , piece_id=5WIVAQ5RHV4RQSLY4B6OQXWMLOJINATJ5BUOD4TGINNJZEPYGSSQ, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.593725098 (456h13m50.36140851s) [noise=true , address=64.49.13.169:4709 , piece_id=X2RC4JCIGJBOPDYEDS7TAFT3LDAOHDAC6AHE75I324UKUAGSIPBA, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.594062759 (456h13m50.361746188s) [noise=true , address=45.91.116.13:4166 , piece_id=OBFIPIYC6XGXQCI63QWEK7I2DOQFKEXDVUZEOLU55BUR5JVVHYRQ, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.594141884 (456h13m50.361825412s) [noise=true , address=45.91.116.12:4235 , piece_id=JXPXLFANCRRVJXOTTCC3ZFE5G4LXVTMA2DSEI7RAPOSMY33ZARAQ, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.594370475 (456h13m50.362053907s) [noise=true , address=45.91.116.7:4035 , piece_id=CYTRHL7GVTG6AALOUDYGNHYXYPMETG37GFG5HZ4RGKVW2622WZ7A, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.594828168 (456h13m50.362511604s) [noise=true , address=77.90.31.12:4603 , piece_id=IEQGTVX6P6PSYLUZJRVHQSLIIGI3OTG76DOAIVLUFUGMH2573K4Q, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.594839192 (456h13m50.362522628s) [noise=true , address=77.90.31.21:4903 , piece_id=4OODH6ZO7XR2S2KDSPBYHOGTER6ZXAZGCJIDK57O2CHZWAHEA4XA, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.595279515 (456h13m50.362962944s) [noise=true , address=77.90.31.17:4785 , piece_id=X53FN5GN3TS3WQ7DJDNHGZRC3MO6AO6OCCF2D73MSPX4XFP243DA, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.595525686 (456h13m50.363209121s) [noise=true , address=45.91.116.5:4304 , piece_id=RDCUSPM6ZIAK2XKALHNBAI2BD7JM4QFFJPUINNXF7NVVI7P5UOKQ, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.595756259 (456h13m50.363439693s) [noise=true , address=77.90.31.13:4645 , piece_id=5CQ6UJBI5U3DE6B5PUCIRM67K6OCSIC5AL7UCADRLTRXJOYBY2EA, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.596197831 (456h13m50.363881266s) [noise=true , address=77.90.31.19:4849 , piece_id=2FA2SN4C6NF2XOUB455BWX4NNW4NDASZXNZWBZQF6BH5K76UA3TQ, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.596267567 (456h13m50.363951s) [noise=true , address=77.90.31.11:4574 , piece_id=CDLA3KF2EAPGWO4PB7O3GVZOGGXOJCRA6YT4UO2T4OHJACACYHPQ, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.596919566 (456h13m50.364603002s) [noise=true , address=45.91.116.10:4114 , piece_id=ZBCHT2CKSZT7VAUHLIE5CZDDWAQGTOCXF3SGJRPHFMM6BHZSAQVA, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.597659197 (456h13m50.365342635s) [noise=true , address=64.49.13.137:4493 , piece_id=25FG6JMLYXV65LDQBL6WJRNNL3VQGSZPLRQ6ZS5Y2FDL23GHKGNA, seg_pos={0 0}, upload=474823]: Uploading piece... Nov 30 10:00:25.660078715 (456h13m50.427762134s) [upload=474823]: Next returned nil segment Nov 30 10:00:25.660090114 (456h13m50.427773539s) [upload=474823]: Waiting for error group managing segments... Nov 30 10:00:25.664319806 (456h13m50.432003232s) [noise=true , address=45.91.116.13:4166 , piece_id=OBFIPIYC6XGXQCI63QWEK7I2DOQFKEXDVUZEOLU55BUR5JVVHYRQ, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:25.664535250 (456h13m50.432218675s) [noise=true , address=45.91.116.5:4304 , piece_id=RDCUSPM6ZIAK2XKALHNBAI2BD7JM4QFFJPUINNXF7NVVI7P5UOKQ, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:25.664852822 (456h13m50.432536247s) [noise=true , address=45.91.116.6:4005 , piece_id=5WIVAQ5RHV4RQSLY4B6OQXWMLOJINATJ5BUOD4TGINNJZEPYGSSQ, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:25.665047792 (456h13m50.432731217s) [noise=true , address=45.91.116.12:4235 , piece_id=JXPXLFANCRRVJXOTTCC3ZFE5G4LXVTMA2DSEI7RAPOSMY33ZARAQ, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:25.665218082 (456h13m50.432901508s) [noise=true , address=45.91.116.7:4035 , piece_id=CYTRHL7GVTG6AALOUDYGNHYXYPMETG37GFG5HZ4RGKVW2622WZ7A, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:25.668059215 (456h13m50.43574264s) [noise=true , address=145.79.227.3:4183 , piece_id=KINVB5A5IEEL5RRBQI62WVL3MCYLV5RLWGI4RSKZTMB76RRD6YVQ, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:25.668734274 (456h13m50.436417699s) [noise=true , address=145.79.227.7:4311 , piece_id=ZNNFEWO24LQSQDGHQPDPPFJOU3JNXRV7PPNQQHVQWKEO4CNCMJXA, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:25.677016764 (456h13m50.444700193s) [noise=true , address=45.91.116.14:4214 , piece_id=5UDQZTYWNVXVSLCYVRD5B7LENDPXEPKIG7AGDHVWUCBN2SVCODCA, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:25.680560413 (456h13m50.448243842s) [noise=true , address=45.91.116.10:4114 , piece_id=ZBCHT2CKSZT7VAUHLIE5CZDDWAQGTOCXF3SGJRPHFMM6BHZSAQVA, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:26.176974310 (456h13m50.944657736s) [noise=true , address=64.49.13.26:4955 , piece_id=QS27N72UQ2KYYPZPA6IP3433GT5QLCGCK4Z6DPLGISRM427A457A, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:26.247495777 (456h13m51.015179205s) [noise=true , address=64.49.13.137:4493 , piece_id=25FG6JMLYXV65LDQBL6WJRNNL3VQGSZPLRQ6ZS5Y2FDL23GHKGNA, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:00:27.476873968 (456h13m52.244557393s) [noise=true , address=64.49.13.169:4709 , piece_id=X2RC4JCIGJBOPDYEDS7TAFT3LDAOHDAC6AHE75I324UKUAGSIPBA, seg_pos={0 0}, upload=474823]: Done uploading piece. err: <nil> Nov 30 10:01:25.530606224 (456h14m50.298289652s) [noise=true , address=77.90.31.19:4849 , piece_id=2FA2SN4C6NF2XOUB455BWX4NNW4NDASZXNZWBZQF6BH5K76UA3TQ, seg_pos={0 0}, upload=474823]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: rpc: dial tcp 77.90.31.19:4849: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.530649997 (456h14m50.298333431s) [noise=true , address=77.90.31.12:4603 , piece_id=IEQGTVX6P6PSYLUZJRVHQSLIIGI3OTG76DOAIVLUFUGMH2573K4Q, seg_pos={0 0}, upload=474823]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.531001629 (456h14m50.298685058s) [noise=true , address=77.90.31.17:4785 , piece_id=X53FN5GN3TS3WQ7DJDNHGZRC3MO6AO6OCCF2D73MSPX4XFP243DA, seg_pos={0 0}, upload=474823]: Done uploading piece. err: ecclient: upload failed (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN, address:77.90.31.17:4785): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:25.531261101 (456h14m50.298944533s) [noise=true , address=77.90.31.13:4645 , piece_id=5CQ6UJBI5U3DE6B5PUCIRM67K6OCSIC5AL7UCADRLTRXJOYBY2EA, seg_pos={0 0}, upload=474823]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 159.223.143.39:47180->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.531273924 (456h14m50.298957363s) [noise=true , address=77.90.31.11:4574 , piece_id=CDLA3KF2EAPGWO4PB7O3GVZOGGXOJCRA6YT4UO2T4OHJACACYHPQ, seg_pos={0 0}, upload=474823]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:58266->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:25.531452521 (456h14m50.299135956s) [noise=true , address=77.90.31.21:4903 , piece_id=4OODH6ZO7XR2S2KDSPBYHOGTER6ZXAZGCJIDK57O2CHZWAHEA4XA, seg_pos={0 0}, upload=474823]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:25.531501332 (456h14m50.299184759s) [seg_pos={0 0}, upload=474823]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 30 10:01:25.549677011 (456h14m50.317360436s) [upload=474823]: Done uploading segments... Nov 30 10:01:25.549727799 (456h14m50.317411233s) [upload=474823]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.020929926s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.001169342s trace: https://jaeger.dev.storj.io/trace/d79b0cf2459300b FAIL (2025-11-30 10:00:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001182398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......................................................XXXXX] duration: 5.000910636s trace: https://jaeger.dev.storj.io/trace/70ade75a86add6ea FAIL (2025-11-30 10:00:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000930489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.024293675s trace: https://jaeger.dev.storj.io/trace/2990adbb2043d1f FAIL (2025-11-30 10:00:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Nov 30 09:59:25.316358210 (456h12m50.084041644s) [upload=474807]: Starting upload Nov 30 09:59:25.316411393 (456h12m50.084094818s) [upload=474807]: Done starting upload Nov 30 09:59:25.316546663 (456h12m50.084230078s) [upload=474807]: Uploading segments... Nov 30 09:59:25.316699393 (456h12m50.084382818s) [upload=474807]: Got next segment. Inline: false Nov 30 09:59:25.316753545 (456h12m50.08443697s) [upload=474807]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 09:59:25.316763647 (456h12m50.084447063s) [upload=474807]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 09:59:25.343545956 (456h12m50.111229392s) [seg_pos={0 0}, upload=474807]: Begin upload segment... Nov 30 09:59:25.343653484 (456h12m50.111336911s) [seg_pos={0 0}, upload=474807]: Done begin upload segment. Nov 30 09:59:25.343718016 (456h12m50.111401451s) [noise=true , address=77.90.31.13:4625 , piece_id=F2M433AAKP4EKRVJXJZZTAQKKKEF4SJHWDQNSY2A2R4TXEWSWTHQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.343737472 (456h12m50.111420954s) [noise=true , address=45.91.116.14:4206 , piece_id=4OWHH3X6SK6LEI24C6FQTFIXMLDM74473US4RKMOK6LMF6FD4EAQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.343823261 (456h12m50.111506691s) [noise=true , address=77.90.31.16:4766 , piece_id=3N67WSRLAABCUTN47NRVMG73MTTMHXTHA62M2LT6B52TA24ZDVVA, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.344427888 (456h12m50.112111318s) [noise=true , address=77.90.31.17:4796 , piece_id=RXS3JEDDWTOVKTUYNOPRKIE6FXJZJJLI7QCRAYSFFLMFOOZBR2AQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.344536539 (456h12m50.112219971s) [noise=true , address=64.49.13.169:4716 , piece_id=X6MVI6WIB6U6QAVM6ACMAA25OZ4IYQVNGGQYWU37RWQUJKOPF5UA, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.345611614 (456h12m50.113295048s) [noise=true , address=77.90.31.12:4621 , piece_id=I3ZYDNIFYJ4ZCRIZWHA53IEZKQ223ND6RKZPAMH3QXLYSM4T347A, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.346279328 (456h12m50.113962762s) [noise=true , address=77.90.31.11:4587 , piece_id=WSKCMWCSJ4ADFMYGEB4VKHFFFGJD7BJFRAXDIMFQFFAPUEBFMSAA, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.346448179 (456h12m50.114131611s) [noise=true , address=77.90.31.10:4543 , piece_id=PKCV7CT5OHQNO4UO6FUIYLH6WUZLKP63ZWZOCLAB56WV7UE5SGUQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.347239737 (456h12m50.114923172s) [noise=true , address=64.49.13.154:4935 , piece_id=PFAZIJ5YK6PNWGIKYXXQ3XJVLC7HH2JHMFKPI77MEYYHBSAYXWPA, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.349260164 (456h12m50.116943601s) [noise=true , address=45.91.116.12:4241 , piece_id=PK2S4DLXDD24EB62PPJN4QYKNPO4TUGLGJNR5NVV47VGXKS3KMXA, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.350490275 (456h12m50.118173711s) [noise=true , address=64.49.13.170:4724 , piece_id=R7TXBJA7ERKHEMLZRUQF5R6GHIO6AU6SBDG2MRCTA53AOE7SLXYQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.351867383 (456h12m50.119550819s) [noise=true , address=45.91.116.5:4320 , piece_id=7SLPTVITZ43MGZVX7MCR7ZKA3SMHUJG6HOPJXEFBGVTCF326WADQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.352683166 (456h12m50.120366597s) [noise=true , address=45.91.116.6:4977 , piece_id=PAYEW5K4HP7XHNSRFQTNDTCAZAZSHKHMPJ3LVDRKRZZM3CP7BWBA, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.356001388 (456h12m50.123684823s) [noise=true , address=145.79.227.6:4288 , piece_id=3U6ZOXVHVZAB72SLQL4OCFX45OGKN3XGRGZRJQSPYVS6ONT6VLKQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.358294752 (456h12m50.125978189s) [noise=true , address=45.91.116.10:4092 , piece_id=62KMOEPES3KRG27JV7CH7PDTDPRACB6IPH2DK72SV4UTDXUDD3IA, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.360366906 (456h12m50.128050341s) [noise=true , address=145.79.227.3:4166 , piece_id=RI63MMTRBOCIYJSWVZCXS6IVVOR5DQUE4NZO67VBP2NVDJ7KJ6NQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.362432355 (456h12m50.130115791s) [noise=true , address=45.91.116.4:4965 , piece_id=WDEXLVZWD2UY4ROYDJMNKVBKUBEHOB4KZIX457V5YHG3I2RSWA4A, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.363135765 (456h12m50.130819199s) [noise=true , address=45.91.116.11:4130 , piece_id=E6QQ3AM2INTVMQ2ECWNACCIDSI2HIDG6H35GVW7Y5KM64W53AKAQ, seg_pos={0 0}, upload=474807]: Uploading piece... Nov 30 09:59:25.411665626 (456h12m50.179349051s) [upload=474807]: Next returned nil segment Nov 30 09:59:25.411677746 (456h12m50.179361175s) [upload=474807]: Waiting for error group managing segments... Nov 30 09:59:25.415293541 (456h12m50.182976966s) [noise=true , address=45.91.116.10:4092 , piece_id=62KMOEPES3KRG27JV7CH7PDTDPRACB6IPH2DK72SV4UTDXUDD3IA, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.415684683 (456h12m50.183368109s) [noise=true , address=45.91.116.5:4320 , piece_id=7SLPTVITZ43MGZVX7MCR7ZKA3SMHUJG6HOPJXEFBGVTCF326WADQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.416129125 (456h12m50.183812539s) [noise=true , address=45.91.116.14:4206 , piece_id=4OWHH3X6SK6LEI24C6FQTFIXMLDM74473US4RKMOK6LMF6FD4EAQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.416276310 (456h12m50.183959725s) [noise=true , address=45.91.116.11:4130 , piece_id=E6QQ3AM2INTVMQ2ECWNACCIDSI2HIDG6H35GVW7Y5KM64W53AKAQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.416613563 (456h12m50.184296988s) [noise=true , address=45.91.116.6:4977 , piece_id=PAYEW5K4HP7XHNSRFQTNDTCAZAZSHKHMPJ3LVDRKRZZM3CP7BWBA, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.416713380 (456h12m50.184396805s) [noise=true , address=45.91.116.12:4241 , piece_id=PK2S4DLXDD24EB62PPJN4QYKNPO4TUGLGJNR5NVV47VGXKS3KMXA, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.416967853 (456h12m50.184651278s) [noise=true , address=45.91.116.4:4965 , piece_id=WDEXLVZWD2UY4ROYDJMNKVBKUBEHOB4KZIX457V5YHG3I2RSWA4A, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.419489215 (456h12m50.187172639s) [noise=true , address=145.79.227.6:4288 , piece_id=3U6ZOXVHVZAB72SLQL4OCFX45OGKN3XGRGZRJQSPYVS6ONT6VLKQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.435942585 (456h12m50.20362601s) [noise=true , address=145.79.227.3:4166 , piece_id=RI63MMTRBOCIYJSWVZCXS6IVVOR5DQUE4NZO67VBP2NVDJ7KJ6NQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.976394734 (456h12m50.744078145s) [noise=true , address=64.49.13.154:4935 , piece_id=PFAZIJ5YK6PNWGIKYXXQ3XJVLC7HH2JHMFKPI77MEYYHBSAYXWPA, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.989112064 (456h12m50.75679549s) [noise=true , address=64.49.13.169:4716 , piece_id=X6MVI6WIB6U6QAVM6ACMAA25OZ4IYQVNGGQYWU37RWQUJKOPF5UA, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 09:59:25.989131339 (456h12m50.756814793s) [noise=true , address=64.49.13.170:4724 , piece_id=R7TXBJA7ERKHEMLZRUQF5R6GHIO6AU6SBDG2MRCTA53AOE7SLXYQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: <nil> Nov 30 10:00:25.285926269 (456h13m50.053609703s) [noise=true , address=77.90.31.10:4543 , piece_id=PKCV7CT5OHQNO4UO6FUIYLH6WUZLKP63ZWZOCLAB56WV7UE5SGUQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:25.286127871 (456h13m50.053811305s) [noise=true , address=77.90.31.17:4796 , piece_id=RXS3JEDDWTOVKTUYNOPRKIE6FXJZJJLI7QCRAYSFFLMFOOZBR2AQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: rpc: dial tcp 77.90.31.17:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:25.286284867 (456h13m50.053968298s) [noise=true , address=77.90.31.13:4625 , piece_id=F2M433AAKP4EKRVJXJZZTAQKKKEF4SJHWDQNSY2A2R4TXEWSWTHQ, seg_pos={0 0}, upload=474807]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: rpc: dial tcp 77.90.31.13:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:25.286472255 (456h13m50.054155684s) [noise=true , address=77.90.31.16:4766 , piece_id=3N67WSRLAABCUTN47NRVMG73MTTMHXTHA62M2LT6B52TA24ZDVVA, seg_pos={0 0}, upload=474807]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 159.223.143.39:50638->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:25.286787038 (456h13m50.054470473s) [noise=true , address=77.90.31.11:4587 , piece_id=WSKCMWCSJ4ADFMYGEB4VKHFFFGJD7BJFRAXDIMFQFFAPUEBFMSAA, seg_pos={0 0}, upload=474807]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: rpc: dial tcp 77.90.31.11:4587: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:25.287044261 (456h13m50.054727692s) [noise=true , address=77.90.31.12:4621 , piece_id=I3ZYDNIFYJ4ZCRIZWHA53IEZKQ223ND6RKZPAMH3QXLYSM4T347A, seg_pos={0 0}, upload=474807]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: write tcp 159.223.143.39:47290->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:25.287080511 (456h13m50.054763928s) [seg_pos={0 0}, upload=474807]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 30 10:00:25.307432383 (456h13m50.075115817s) [upload=474807]: Done uploading segments... Nov 30 10:00:25.307499551 (456h13m50.075182981s) [upload=474807]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.024308787s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.024329115s trace: https://jaeger.dev.storj.io/trace/2fd1791b5208ba93 FAIL (2025-11-30 10:00:23 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== .164.143:4700 , piece_id=JGOZQ4DCYULJ6IVJJFGVZJO5TDL5T4YS7L7JL54KYWJM7K3RLR7A, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.228441758 (456h12m47.996125186s) [noise=true , address=145.79.227.17:4001 , piece_id=H2N7DSDEIU3P47D6EVUFF3GKJ5G7E4N7AWLPFNKAPJX6AL5IMKWA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.231434659 (456h12m47.999118095s) [noise=true , address=64.49.13.139:4469 , piece_id=4Y6C2UMJZUW3GBXDHVHW4FVIW7HGNP4PAMBNR4OJO7TNZZLNKPJQ, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.232121962 (456h12m47.99980539s) [noise=true , address=77.90.31.19:4871 , piece_id=S7ACWW5CJ5NBREWINT5KOL2U7XQN353ARRDK43VR5PNRTNIU6GUA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.232731096 (456h12m48.000414524s) [noise=true , address=77.90.31.21:4894 , piece_id=IKS4P7CBP27ILIZ66EIABOIBD7CM2M3R6PGECISCRWW6EG2MM2CA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.233243772 (456h12m48.0009272s) [noise=true , address=145.79.227.2:4132 , piece_id=DFWQO3UH7PTQPKMVD6LPVWMJQ2KHDXSCBG6XAH5IDU46YSDYYLFQ, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.233784642 (456h12m48.001468073s) [noise=true , address=77.90.31.15:4701 , piece_id=7PBGEXVNWMFJ4NJHKF3S7YS3BTEPARCJMLAFQSA4U25QJE7UR77A, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.235871135 (456h12m48.003554563s) [noise=true , address=145.79.227.4:4202 , piece_id=UYNFXPP4I6YW4GBKUG7SPXMDZHI3NCKNE5CHQSDDMJR6SY5N4NHA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.237632529 (456h12m48.005315963s) [noise=true , address=77.90.31.13:4632 , piece_id=R5WJ5PI67EIY5OC5FAOE2N7KTM3DAHK4ZAHU5SGD2HUDKDLQTGMA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.244709926 (456h12m48.01239336s) [noise=true , address=45.91.116.6:4981 , piece_id=5EIRWR4T45CIMJZ76K3ZS3EQJKXPX53QW7VTTFEB5CQTEJ2YT66Q, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.249134387 (456h12m48.016817811s) [noise=true , address=64.49.13.169:4705 , piece_id=7LADBZS4UOARHH3TP4HEYUULWBNGCGKGSWMQQX6ZIBTPT6AUGBKQ, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.252596901 (456h12m48.020326356s) [noise=true , address=77.90.31.14:4670 , piece_id=UNMEM5JN37XPCPVQJXX52TD4EQOKIN6OHQ2IPPGNBDEEM3ZXW7PA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.254046663 (456h12m48.021730097s) [noise=true , address=77.90.31.12:4607 , piece_id=HDANYNV4LICNTWXMWLWI6EHAA6BFXXNUNYHFLYCAP4MEMFMF4EWA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.254796487 (456h12m48.022479918s) [noise=true , address=64.49.13.27:4967 , piece_id=DTBFNSQ7QOT6AW7QOZ6IJJ6G2XFXDJULDENI3XW5OAJK5JLPKPBA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.257877619 (456h12m48.025561048s) [noise=true , address=77.90.31.16:4750 , piece_id=VBT2RBIPG5GSDCHZV7ELUF2U7XWBTGCX7JC2XYE6ZT53QV6CDT2A, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.259896751 (456h12m48.027580194s) [noise=true , address=77.90.31.18:4837 , piece_id=MMY654NBCALSFKT5FUXRGFGH6KAHAVZ4MH3DLVAQBVN7EFEQ2JRA, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.260626623 (456h12m48.028310054s) [noise=true , address=77.90.31.17:4778 , piece_id=L5663WOPIG3DCL6YJNOHXIGEQUZZOVU4JKQR3YB34A2WLOM3TULQ, seg_pos={0 0}, upload=474805]: Uploading piece... Nov 30 09:59:23.296995629 (456h12m48.064679061s) [upload=474805]: Next returned nil segment Nov 30 09:59:23.297011245 (456h12m48.064694676s) [upload=474805]: Waiting for error group managing segments... Nov 30 09:59:23.301205403 (456h12m48.06888883s) [noise=true , address=45.91.116.6:4981 , piece_id=5EIRWR4T45CIMJZ76K3ZS3EQJKXPX53QW7VTTFEB5CQTEJ2YT66Q, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 09:59:23.304457683 (456h12m48.072141108s) [noise=true , address=145.79.227.4:4202 , piece_id=UYNFXPP4I6YW4GBKUG7SPXMDZHI3NCKNE5CHQSDDMJR6SY5N4NHA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 09:59:23.304855197 (456h12m48.072538625s) [noise=true , address=145.79.227.2:4132 , piece_id=DFWQO3UH7PTQPKMVD6LPVWMJQ2KHDXSCBG6XAH5IDU46YSDYYLFQ, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 09:59:23.305169692 (456h12m48.072853124s) [noise=true , address=145.79.227.17:4001 , piece_id=H2N7DSDEIU3P47D6EVUFF3GKJ5G7E4N7AWLPFNKAPJX6AL5IMKWA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 09:59:23.828726244 (456h12m48.596409669s) [noise=true , address=64.49.13.139:4469 , piece_id=4Y6C2UMJZUW3GBXDHVHW4FVIW7HGNP4PAMBNR4OJO7TNZZLNKPJQ, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 09:59:23.836497904 (456h12m48.604181329s) [noise=true , address=193.149.164.143:4700 , piece_id=JGOZQ4DCYULJ6IVJJFGVZJO5TDL5T4YS7L7JL54KYWJM7K3RLR7A, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 09:59:23.877055844 (456h12m48.64473927s) [noise=true , address=64.49.13.154:4931 , piece_id=VLCHP24XZRATYBXAVTCX4G3QAXDTC5IDD7VFTLXJDITWUS3WNB2Q, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 09:59:23.914494814 (456h12m48.682178246s) [noise=true , address=64.49.13.27:4967 , piece_id=DTBFNSQ7QOT6AW7QOZ6IJJ6G2XFXDJULDENI3XW5OAJK5JLPKPBA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 09:59:24.339690830 (456h12m49.107374264s) [noise=true , address=64.49.13.169:4705 , piece_id=7LADBZS4UOARHH3TP4HEYUULWBNGCGKGSWMQQX6ZIBTPT6AUGBKQ, seg_pos={0 0}, upload=474805]: Done uploading piece. err: <nil> Nov 30 10:00:23.165814602 (456h13m47.933498041s) [noise=true , address=77.90.31.18:4837 , piece_id=MMY654NBCALSFKT5FUXRGFGH6KAHAVZ4MH3DLVAQBVN7EFEQ2JRA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: rpc: dial tcp 77.90.31.18:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.165854487 (456h13m47.933537922s) [noise=true , address=77.90.31.19:4871 , piece_id=S7ACWW5CJ5NBREWINT5KOL2U7XQN353ARRDK43VR5PNRTNIU6GUA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.166155543 (456h13m47.933838968s) [noise=true , address=77.90.31.14:4670 , piece_id=UNMEM5JN37XPCPVQJXX52TD4EQOKIN6OHQ2IPPGNBDEEM3ZXW7PA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:12LQ2JYC2GB67iqFPW69DyASSKH9yN558jGYBFd2cRN8abegiTE, address:77.90.31.14:4670): protocol: rpc: dial tcp 77.90.31.14:4670: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.166444968 (456h13m47.934128413s) [noise=true , address=77.90.31.13:4632 , piece_id=R5WJ5PI67EIY5OC5FAOE2N7KTM3DAHK4ZAHU5SGD2HUDKDLQTGMA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.166701674 (456h13m47.934385112s) [noise=true , address=77.90.31.21:4894 , piece_id=IKS4P7CBP27ILIZ66EIABOIBD7CM2M3R6PGECISCRWW6EG2MM2CA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG, address:77.90.31.21:4894): protocol: rpc: dial tcp 77.90.31.21:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.166940247 (456h13m47.934623679s) [noise=true , address=77.90.31.17:4778 , piece_id=L5663WOPIG3DCL6YJNOHXIGEQUZZOVU4JKQR3YB34A2WLOM3TULQ, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.166990088 (456h13m47.934673523s) [noise=true , address=77.90.31.15:4701 , piece_id=7PBGEXVNWMFJ4NJHKF3S7YS3BTEPARCJMLAFQSA4U25QJE7UR77A, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.167268896 (456h13m47.934952321s) [noise=true , address=77.90.31.12:4607 , piece_id=HDANYNV4LICNTWXMWLWI6EHAA6BFXXNUNYHFLYCAP4MEMFMF4EWA, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.167387839 (456h13m47.935071256s) [noise=true , address=77.90.31.16:4750 , piece_id=VBT2RBIPG5GSDCHZV7ELUF2U7XWBTGCX7JC2XYE6ZT53QV6CDT2A, seg_pos={0 0}, upload=474805]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:23.167416914 (456h13m47.935100338s) [seg_pos={0 0}, upload=474805]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Nov 30 10:00:23.187552335 (456h13m47.955235772s) [upload=474805]: Done uploading segments... Nov 30 10:00:23.187613818 (456h13m47.955297254s) [upload=474805]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.024346504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.00152251s trace: https://jaeger.dev.storj.io/trace/71a2d5ade7f7bcb1 FAIL (2025-11-30 09:59:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.0015801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................................................XXXX] duration: 5.000786477s trace: https://jaeger.dev.storj.io/trace/4c3ada4324e1b377 FAIL (2025-11-30 09:59:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00088035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.024836359s trace: https://jaeger.dev.storj.io/trace/25110cb629bd481e FAIL (2025-11-30 09:59:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Nov 30 09:58:25.252376749 (456h11m50.02006018s) [upload=474790]: Starting upload Nov 30 09:58:25.252455387 (456h11m50.020138821s) [upload=474790]: Done starting upload Nov 30 09:58:25.252574107 (456h11m50.020257523s) [upload=474790]: Uploading segments... Nov 30 09:58:25.252958607 (456h11m50.020642032s) [upload=474790]: Got next segment. Inline: false Nov 30 09:58:25.253005764 (456h11m50.020689184s) [upload=474790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 09:58:25.253012729 (456h11m50.020696161s) [upload=474790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 09:58:25.279633988 (456h11m50.047317415s) [seg_pos={0 0}, upload=474790]: Begin upload segment... Nov 30 09:58:25.279685724 (456h11m50.047369139s) [seg_pos={0 0}, upload=474790]: Done begin upload segment. Nov 30 09:58:25.279714866 (456h11m50.047398294s) [noise=true , address=45.91.116.7:4040 , piece_id=MDASZIIHKNW6UY7GGTC3WMOHQOTBSV4NKTLM4OHUKFYBVBXFOFYQ, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.279792224 (456h11m50.047475658s) [noise=true , address=45.91.116.9:4275 , piece_id=KGXYJC3BPG6II44K7AKOEFARCZYVQXXZHKBAVGEAJ527NR6BHDEA, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.280412609 (456h11m50.048096044s) [noise=true , address=77.90.31.19:4854 , piece_id=VXGWCWL7I7464GMQYAGIZEDRR7EHEWKRJ3ME7PR3NSKXXVGCZ3KA, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.280935233 (456h11m50.048618659s) [noise=true , address=77.90.31.11:4572 , piece_id=VPSEUEKZGTJAFMUVHHSSI4WQSZUK54XUXJ3RPM7CV4ZBE7P7MF3Q, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.281203175 (456h11m50.048886606s) [noise=true , address=45.91.116.8:4053 , piece_id=CZZEGL2XLERYS3HR6BB6EUNYVOHEBIDC2IOH7RANMOSI6V7F34KA, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.281317341 (456h11m50.049000767s) [noise=true , address=77.90.31.17:4785 , piece_id=CKT7QVDR42Y4AJBTU3VELTEJUPZJ5A6UKQ3ILCEV56XLBDZYQVZA, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.282583068 (456h11m50.050266501s) [noise=true , address=77.90.31.14:4690 , piece_id=SCFJ5VNZJAUZCUJLZXZSPUQJHELQO4766FGTHJZRMEBOCUOTDSXA, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.283131014 (456h11m50.050814439s) [noise=true , address=45.91.116.5:4331 , piece_id=CNHY4Y34XGIGFF5LOMALHUTXKUMQRHV35KFUTW6L5FR63RBLCEJQ, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.283746928 (456h11m50.051430356s) [noise=true , address=64.49.13.169:4709 , piece_id=AYLJ3U3ZMAM4GAONWB2OVRMHQM3UYITD4NQRMA6WLWXUNIMVCKWQ, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.284156043 (456h11m50.051839469s) [noise=true , address=145.79.227.40:4055 , piece_id=FMTAPHUGSGXVN4KR6DVSXZXKG4KUIVBG65IWDED5AFGS6QXPSWGA, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.284486708 (456h11m50.052170142s) [noise=true , address=45.91.116.12:4252 , piece_id=LL7O7GYRC36YHW6KKTCEFMTVDTF77ANMIUOR3I4KSUSM2XHVBASQ, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.285005422 (456h11m50.052688852s) [noise=true , address=45.91.116.13:4185 , piece_id=UT7TMGEEPYOBCB25CM34AXYT5CY6FTKR7OKM7LNZYHWZY5N7C6CQ, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.286218124 (456h11m50.053901559s) [noise=true , address=45.91.116.11:4137 , piece_id=A5777EY5KLVCELUCQT7QV3KYJFIXX6EZC5QAJVRMGP3JZUPDLPWA, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.286277839 (456h11m50.053961265s) [noise=true , address=77.90.31.21:4886 , piece_id=ZYC3OFU2GGGFKYFWTRV7JXX2HAHLQVYRQUWHGGUFVABFNOM4ZAMQ, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.286631846 (456h11m50.054315271s) [noise=true , address=45.91.116.6:4004 , piece_id=HFHF5BWS6GXXEB24VTTO5HKUYY4PFHMPLDJMSMORRGA65SJQWL4A, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.289628091 (456h11m50.057311516s) [noise=true , address=145.79.227.4:4202 , piece_id=FX6JNBA7FNDDHKY7BABMYMUQYHME2XTAKKU7ZLSZ7DQQLAFAA2YQ, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.291631129 (456h11m50.059314557s) [noise=true , address=77.90.31.12:4618 , piece_id=5HRJ25GGHE5MGY6V7DCER2GXXWE2GCDLUEC5PM4ANJB5ZD65QMMA, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.292380937 (456h11m50.060064365s) [noise=true , address=145.79.227.12:4772 , piece_id=YX47OH22RKM6UCQZTKXD2YN6LZM755HS5SBODCEVYBAHUM3GWDSQ, seg_pos={0 0}, upload=474790]: Uploading piece... Nov 30 09:58:25.339003634 (456h11m50.106687047s) [upload=474790]: Next returned nil segment Nov 30 09:58:25.339043014 (456h11m50.106726448s) [upload=474790]: Waiting for error group managing segments... Nov 30 09:58:25.343375563 (456h11m50.111058993s) [noise=true , address=45.91.116.11:4137 , piece_id=A5777EY5KLVCELUCQT7QV3KYJFIXX6EZC5QAJVRMGP3JZUPDLPWA, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.344299376 (456h11m50.111982803s) [noise=true , address=45.91.116.13:4185 , piece_id=UT7TMGEEPYOBCB25CM34AXYT5CY6FTKR7OKM7LNZYHWZY5N7C6CQ, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.344960518 (456h11m50.112643948s) [noise=true , address=45.91.116.8:4053 , piece_id=CZZEGL2XLERYS3HR6BB6EUNYVOHEBIDC2IOH7RANMOSI6V7F34KA, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.346149557 (456h11m50.113832986s) [noise=true , address=45.91.116.7:4040 , piece_id=MDASZIIHKNW6UY7GGTC3WMOHQOTBSV4NKTLM4OHUKFYBVBXFOFYQ, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.346843028 (456h11m50.114526453s) [noise=true , address=45.91.116.6:4004 , piece_id=HFHF5BWS6GXXEB24VTTO5HKUYY4PFHMPLDJMSMORRGA65SJQWL4A, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.347174645 (456h11m50.114858077s) [noise=true , address=45.91.116.9:4275 , piece_id=KGXYJC3BPG6II44K7AKOEFARCZYVQXXZHKBAVGEAJ527NR6BHDEA, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.348088935 (456h11m50.115772361s) [noise=true , address=145.79.227.40:4055 , piece_id=FMTAPHUGSGXVN4KR6DVSXZXKG4KUIVBG65IWDED5AFGS6QXPSWGA, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.348346548 (456h11m50.116029977s) [noise=true , address=145.79.227.4:4202 , piece_id=FX6JNBA7FNDDHKY7BABMYMUQYHME2XTAKKU7ZLSZ7DQQLAFAA2YQ, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.350075703 (456h11m50.117759115s) [noise=true , address=45.91.116.12:4252 , piece_id=LL7O7GYRC36YHW6KKTCEFMTVDTF77ANMIUOR3I4KSUSM2XHVBASQ, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.355776862 (456h11m50.123460295s) [noise=true , address=45.91.116.5:4331 , piece_id=CNHY4Y34XGIGFF5LOMALHUTXKUMQRHV35KFUTW6L5FR63RBLCEJQ, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:25.364872459 (456h11m50.132555885s) [noise=true , address=145.79.227.12:4772 , piece_id=YX47OH22RKM6UCQZTKXD2YN6LZM755HS5SBODCEVYBAHUM3GWDSQ, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:58:27.115598134 (456h11m51.883281565s) [noise=true , address=64.49.13.169:4709 , piece_id=AYLJ3U3ZMAM4GAONWB2OVRMHQM3UYITD4NQRMA6WLWXUNIMVCKWQ, seg_pos={0 0}, upload=474790]: Done uploading piece. err: <nil> Nov 30 09:59:25.217100092 (456h12m49.984783526s) [noise=true , address=77.90.31.17:4785 , piece_id=CKT7QVDR42Y4AJBTU3VELTEJUPZJ5A6UKQ3ILCEV56XLBDZYQVZA, seg_pos={0 0}, upload=474790]: Done uploading piece. err: ecclient: upload failed (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN, address:77.90.31.17:4785): protocol: rpc: dial tcp 77.90.31.17:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:25.217132490 (456h12m49.984815926s) [noise=true , address=77.90.31.21:4886 , piece_id=ZYC3OFU2GGGFKYFWTRV7JXX2HAHLQVYRQUWHGGUFVABFNOM4ZAMQ, seg_pos={0 0}, upload=474790]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: rpc: dial tcp 77.90.31.21:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:25.217412740 (456h12m49.985096182s) [noise=true , address=77.90.31.19:4854 , piece_id=VXGWCWL7I7464GMQYAGIZEDRR7EHEWKRJ3ME7PR3NSKXXVGCZ3KA, seg_pos={0 0}, upload=474790]: Done uploading piece. err: ecclient: upload failed (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE, address:77.90.31.19:4854): protocol: rpc: dial tcp 77.90.31.19:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:25.217432374 (456h12m49.985115802s) [noise=true , address=77.90.31.14:4690 , piece_id=SCFJ5VNZJAUZCUJLZXZSPUQJHELQO4766FGTHJZRMEBOCUOTDSXA, seg_pos={0 0}, upload=474790]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:25.217707394 (456h12m49.985390823s) [noise=true , address=77.90.31.11:4572 , piece_id=VPSEUEKZGTJAFMUVHHSSI4WQSZUK54XUXJ3RPM7CV4ZBE7P7MF3Q, seg_pos={0 0}, upload=474790]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: write tcp 159.223.143.39:44916->77.90.31.11:4572: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:25.217863091 (456h12m49.985546509s) [noise=true , address=77.90.31.12:4618 , piece_id=5HRJ25GGHE5MGY6V7DCER2GXXWE2GCDLUEC5PM4ANJB5ZD65QMMA, seg_pos={0 0}, upload=474790]: Done uploading piece. err: ecclient: upload failed (node:1XuRKaSZC9qb9M17ctkcNkf5dSkBYpDBHgxMS56wThuw8s2R6, address:77.90.31.12:4618): protocol: rpc: dial tcp 77.90.31.12:4618: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:25.217911316 (456h12m49.98559473s) [seg_pos={0 0}, upload=474790]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 30 09:59:25.239111410 (456h12m50.006794838s) [upload=474790]: Done uploading segments... Nov 30 09:59:25.239149634 (456h12m50.006833063s) [upload=474790]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.024860492s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................................XXX] duration: 5.000426719s trace: https://jaeger.dev.storj.io/trace/465035dcd2adaeb6 FAIL (2025-11-30 09:59:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000451083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.024272725s trace: https://jaeger.dev.storj.io/trace/5da9000648500c4e FAIL (2025-11-30 09:59:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ng piece... Nov 30 09:58:23.145428017 (456h11m47.913111443s) [noise=true , address=77.90.31.14:4694 , piece_id=HBIB6IPNSH3P72WANCZUW6ZWDKDSVU5V6QSBA4AYVCXC2WRYC3KQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.145975602 (456h11m47.913659032s) [noise=true , address=45.91.116.8:4053 , piece_id=6543UMEAOC6GOZPUKFDTYA7NMQ2IGCROOLQYRRXCOKOEU2RBJXDQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.147096930 (456h11m47.914780365s) [noise=true , address=77.90.31.18:4812 , piece_id=MVTBLZHP7YAACIORW5MLS37PMCJLTIGTEDEIPZUTHXP4A77UDG7Q, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.147290397 (456h11m47.914973821s) [noise=true , address=77.90.31.12:4606 , piece_id=FPWLECQSYHSM3INKVGBW3JR3643KWCTWTU4LSD3QC7BRIGIV5GWQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.147784694 (456h11m47.915468125s) [noise=true , address=77.90.31.17:4783 , piece_id=C7Z4I2QFIV6Q3FXAJBQ7WHHXNNYBIK2DQPY2KU2GIRPRNIIOL73Q, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.147843913 (456h11m47.915527338s) [noise=true , address=77.90.31.21:4898 , piece_id=GMNWS6I3RIA2EPKV4FAANFWOULWEKZZZQFCSNWQZ3NWI7O2J3VWQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.148494396 (456h11m47.916177822s) [noise=true , address=45.91.116.13:4169 , piece_id=DG5R4VMWFUS63G3SLZH6EA5RXB6DBTREI43QZUSVEQYSXB33VLJQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.148610635 (456h11m47.916294066s) [noise=true , address=77.90.31.11:4559 , piece_id=YBDBB4TGTCIBYMBRTOC33U3OHTPWZQEAZASISZFNYRRPI7FGY6VQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.148906492 (456h11m47.91658993s) [noise=true , address=45.91.116.5:4328 , piece_id=QAA6DT7ZVLBMBOJRGXCJLYQXINMGZBTJWRZZPVRQMA3LEXPHT5KQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.149381236 (456h11m47.917064663s) [noise=true , address=145.79.227.19:4109 , piece_id=6MYB26SX2PH7QLBGQQE3JN3IH4T6GFGQHHO3SRCNJYYNIL5QDPZA, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.149985023 (456h11m47.917668454s) [noise=true , address=45.91.116.6:4985 , piece_id=OASLHCZRD4BXK7RHBVCA4742FSKOAV3OQRK7DTWCB5JW3GGGHINQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.150359300 (456h11m47.918042728s) [noise=true , address=77.90.31.15:4707 , piece_id=AFJD6UXQ3OTBTIP2OIKPWRZUW2M52RELW4O5NUDDO34YMNUGJZHQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.150956364 (456h11m47.918639795s) [noise=true , address=77.90.31.16:4739 , piece_id=X2LLEGFAVIOZN6XBNGFRHHLBC55LOXDDJKYF6MJW3W32VH46UEXQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.151427494 (456h11m47.919110928s) [noise=true , address=64.49.13.154:4932 , piece_id=HG452W3KZI4K53TEPFVTG3E3ZHJKZR2OPLTR6SZPBPAOKQJOAORQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.154422472 (456h11m47.922105904s) [noise=true , address=77.90.31.10:4532 , piece_id=OELKRWS2KRRFQVWKPM7YLD3GV6APMGZXSNJGCASUNLFQBHIJQ4EQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.155786221 (456h11m47.923469649s) [noise=true , address=45.91.116.9:4286 , piece_id=H4INRAMPHMVMEU5VXDVJNMYWMVY5AEHMJURY2P6R3ZCOFTMFXWAQ, seg_pos={0 0}, upload=474789]: Uploading piece... Nov 30 09:58:23.231507253 (456h11m47.999190672s) [upload=474789]: Next returned nil segment Nov 30 09:58:23.231543048 (456h11m47.999226483s) [upload=474789]: Waiting for error group managing segments... Nov 30 09:58:23.244454487 (456h11m48.012137913s) [noise=true , address=145.79.227.19:4109 , piece_id=6MYB26SX2PH7QLBGQQE3JN3IH4T6GFGQHHO3SRCNJYYNIL5QDPZA, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:58:23.245132387 (456h11m48.012815813s) [noise=true , address=45.91.116.9:4286 , piece_id=H4INRAMPHMVMEU5VXDVJNMYWMVY5AEHMJURY2P6R3ZCOFTMFXWAQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:58:23.245517079 (456h11m48.013200503s) [noise=true , address=45.91.116.13:4169 , piece_id=DG5R4VMWFUS63G3SLZH6EA5RXB6DBTREI43QZUSVEQYSXB33VLJQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:58:23.245983983 (456h11m48.013667408s) [noise=true , address=45.91.116.12:4252 , piece_id=ABCVVSSKQ5PMGSM32AM765J4NKSK5C2HHALIN56734REX36UK6EQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:58:23.246701023 (456h11m48.014384438s) [noise=true , address=45.91.116.6:4985 , piece_id=OASLHCZRD4BXK7RHBVCA4742FSKOAV3OQRK7DTWCB5JW3GGGHINQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:58:23.247232261 (456h11m48.014915687s) [noise=true , address=45.91.116.8:4053 , piece_id=6543UMEAOC6GOZPUKFDTYA7NMQ2IGCROOLQYRRXCOKOEU2RBJXDQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:58:23.266841685 (456h11m48.034525112s) [noise=true , address=45.91.116.5:4328 , piece_id=QAA6DT7ZVLBMBOJRGXCJLYQXINMGZBTJWRZZPVRQMA3LEXPHT5KQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:58:23.760777156 (456h11m48.528460585s) [noise=true , address=64.49.13.154:4932 , piece_id=HG452W3KZI4K53TEPFVTG3E3ZHJKZR2OPLTR6SZPBPAOKQJOAORQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:58:25.108077294 (456h11m49.875760718s) [noise=true , address=64.49.13.169:4710 , piece_id=GINUB3GXVGZ5TMYWMPX65CPVJ53SS3TJRRKPZSAZUC45PV4R6QHQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: <nil> Nov 30 09:59:23.085551725 (456h12m47.85323516s) [noise=true , address=77.90.31.12:4606 , piece_id=FPWLECQSYHSM3INKVGBW3JR3643KWCTWTU4LSD3QC7BRIGIV5GWQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs, address:77.90.31.12:4606): protocol: rpc: dial tcp 77.90.31.12:4606: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.085588571 (456h12m47.853272006s) [noise=true , address=77.90.31.14:4694 , piece_id=HBIB6IPNSH3P72WANCZUW6ZWDKDSVU5V6QSBA4AYVCXC2WRYC3KQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:12qRMWzWyft2skqPXqUgPp41JY8faKf5K2En25vFdxp5dArE5SV, address:77.90.31.14:4694): protocol: write tcp 159.223.143.39:59966->77.90.31.14:4694: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.085813270 (456h12m47.853496703s) [noise=true , address=77.90.31.17:4783 , piece_id=C7Z4I2QFIV6Q3FXAJBQ7WHHXNNYBIK2DQPY2KU2GIRPRNIIOL73Q, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: rpc: dial tcp 77.90.31.17:4783: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.086017119 (456h12m47.853700546s) [noise=true , address=77.90.31.10:4532 , piece_id=OELKRWS2KRRFQVWKPM7YLD3GV6APMGZXSNJGCASUNLFQBHIJQ4EQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 159.223.143.39:59674->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.086041803 (456h12m47.853725236s) [noise=true , address=77.90.31.11:4559 , piece_id=YBDBB4TGTCIBYMBRTOC33U3OHTPWZQEAZASISZFNYRRPI7FGY6VQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: rpc: dial tcp 77.90.31.11:4559: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.086261207 (456h12m47.853944642s) [noise=true , address=77.90.31.18:4812 , piece_id=MVTBLZHP7YAACIORW5MLS37PMCJLTIGTEDEIPZUTHXP4A77UDG7Q, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.086311494 (456h12m47.853994912s) [noise=true , address=77.90.31.15:4707 , piece_id=AFJD6UXQ3OTBTIP2OIKPWRZUW2M52RELW4O5NUDDO34YMNUGJZHQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: write tcp 159.223.143.39:37808->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.086378688 (456h12m47.854062121s) [noise=true , address=77.90.31.21:4898 , piece_id=GMNWS6I3RIA2EPKV4FAANFWOULWEKZZZQFCSNWQZ3NWI7O2J3VWQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx, address:77.90.31.21:4898): protocol: rpc: dial tcp 77.90.31.21:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.086610134 (456h12m47.854293565s) [noise=true , address=77.90.31.16:4739 , piece_id=X2LLEGFAVIOZN6XBNGFRHHLBC55LOXDDJKYF6MJW3W32VH46UEXQ, seg_pos={0 0}, upload=474789]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: rpc: dial tcp 77.90.31.16:4739: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:23.086653842 (456h12m47.854337257s) [seg_pos={0 0}, upload=474789]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Nov 30 09:59:23.108133566 (456h12m47.875816997s) [upload=474789]: Done uploading segments... Nov 30 09:59:23.108175631 (456h12m47.875859049s) [upload=474789]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.02428975s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001609525s trace: https://jaeger.dev.storj.io/trace/251846b80bcb92b5 FAIL (2025-11-30 09:58:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001630855s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.000867284s trace: https://jaeger.dev.storj.io/trace/33523d1047275f7f FAIL (2025-11-30 09:58:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00088701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.026986322s trace: https://jaeger.dev.storj.io/trace/11072d25f0de8b4b FAIL (2025-11-30 09:58:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== . Nov 30 09:57:23.066512300 (456h10m47.834195729s) [noise=true , address=77.90.31.10:4524 , piece_id=573X3XDKEJGSWY6LD7NRX2UCI4KRXEHQ3JVNU6I2OXE4N2TBTM2A, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.067150806 (456h10m47.83483424s) [noise=true , address=45.91.116.12:4248 , piece_id=PWKHM3QF5NV73E2BGPAESXGQL7EMCJX6PTL3226KPXXTY2HFCJMA, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.067645900 (456h10m47.835329328s) [noise=true , address=45.91.116.8:4067 , piece_id=PTHED75KR4G5GHHAPFD6XYEPEINXVWWWIEFBYDLFWTYSARLNJHZA, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.067963588 (456h10m47.835647018s) [noise=true , address=64.49.13.137:4495 , piece_id=5BID2HXBN532CUD3CGMVCDKLRK5PZ2JM6X2FEZEGFYDEJBZEWKZQ, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.068374856 (456h10m47.836058291s) [noise=true , address=45.91.116.10:4105 , piece_id=OR7DDBCKC2J4OCEEUIPPHS4RNGIBHCN3S5N2INNNICCBFKEHQTVQ, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.068444381 (456h10m47.836127811s) [noise=true , address=45.91.116.6:4982 , piece_id=EUTA22LHLMORZUGBHJRZYEHAIL6CXSB7DLKBDZY56X7DDPCHLJ2A, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.068829586 (456h10m47.836513s) [noise=true , address=145.79.227.14:4860 , piece_id=DTPFVLOAEUKUZAYOTRDTS5NVUHN4NX52TA5AO5IRJPNHFWTPG33A, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.068999734 (456h10m47.836683166s) [noise=true , address=77.90.31.15:4725 , piece_id=5ZCXRVFD3FHULPIH3VRZ4Y2FWNLN3FWM57RQBID77Q2GCPJYVX2Q, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.069231264 (456h10m47.836914696s) [noise=true , address=77.90.31.19:4872 , piece_id=5GYPMXHMRS3F263YJNUEFBSFSUVLXRAQBELHW7QQYNUNOVRM5DEQ, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.069573213 (456h10m47.837256629s) [noise=true , address=45.91.116.9:4271 , piece_id=QWEBJSQGUHNUEAYLH3GOXD7QHPAR5G57DNJLVIOWK3NEB7XAR7FQ, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.069579500 (456h10m47.837262938s) [noise=true , address=77.90.31.13:4659 , piece_id=TJGNIW2P3ZA2RMWBN46Q5CKHNO6UCLXYRFDMI6FT6LFRNHF2ZIEQ, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.069958720 (456h10m47.837642151s) [noise=true , address=145.79.227.7:4300 , piece_id=W7PUG5ER4SLXRBDH2622CGZORIH27ZHD7EKH5KLVSYXKIFGPYPHA, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.070279132 (456h10m47.837962567s) [noise=true , address=77.90.31.11:4569 , piece_id=BICAVERJ7E3KIGZDEQPGN3DJXHI4KC3CKBJX35HKKSUTOXDLW6TA, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.070384639 (456h10m47.83806809s) [noise=true , address=77.90.31.18:4813 , piece_id=WFWS6S5NX2FS63VFLXEIGQ24X4TJU3KPPV3LGJHILZWB6EBC4RUQ, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.070951338 (456h10m47.838634795s) [noise=true , address=64.49.13.169:4711 , piece_id=DK7A6JOWXKKLVH2ARPLKV5MJGIJ2KRLUPV3CKXC45BKOJATREX5Q, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.070967309 (456h10m47.838650739s) [noise=true , address=77.90.31.21:4893 , piece_id=EB4RNEEBH5S72XKGQNNFPWSTE5UKQBNXQLCKMTFFVIHC6AIP6GSQ, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.071512132 (456h10m47.839195557s) [noise=true , address=145.79.227.52:4459 , piece_id=H6ES4GHDRJWGQNHYPQOQQKF6YOT7FH7ZJGMBF3TSGXN5F4L42DJA, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.071975590 (456h10m47.839659023s) [noise=true , address=45.91.116.13:4160 , piece_id=EWF3LPPZIRDMYSZLNEB6NEJ2DPOEBODCGHUIGI3VSVDZONIGHTTA, seg_pos={0 0}, upload=474774]: Uploading piece... Nov 30 09:57:23.127760782 (456h10m47.895444195s) [upload=474774]: Next returned nil segment Nov 30 09:57:23.127773091 (456h10m47.895456517s) [upload=474774]: Waiting for error group managing segments... Nov 30 09:57:23.130775988 (456h10m47.898459414s) [noise=true , address=45.91.116.10:4105 , piece_id=OR7DDBCKC2J4OCEEUIPPHS4RNGIBHCN3S5N2INNNICCBFKEHQTVQ, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.131221055 (456h10m47.898904481s) [noise=true , address=45.91.116.6:4982 , piece_id=EUTA22LHLMORZUGBHJRZYEHAIL6CXSB7DLKBDZY56X7DDPCHLJ2A, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.131821396 (456h10m47.899504821s) [noise=true , address=45.91.116.8:4067 , piece_id=PTHED75KR4G5GHHAPFD6XYEPEINXVWWWIEFBYDLFWTYSARLNJHZA, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.131869956 (456h10m47.899553384s) [noise=true , address=45.91.116.9:4271 , piece_id=QWEBJSQGUHNUEAYLH3GOXD7QHPAR5G57DNJLVIOWK3NEB7XAR7FQ, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.135094254 (456h10m47.902777682s) [noise=true , address=145.79.227.52:4459 , piece_id=H6ES4GHDRJWGQNHYPQOQQKF6YOT7FH7ZJGMBF3TSGXN5F4L42DJA, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.135405652 (456h10m47.903089078s) [noise=true , address=145.79.227.7:4300 , piece_id=W7PUG5ER4SLXRBDH2622CGZORIH27ZHD7EKH5KLVSYXKIFGPYPHA, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.136098497 (456h10m47.903781924s) [noise=true , address=145.79.227.14:4860 , piece_id=DTPFVLOAEUKUZAYOTRDTS5NVUHN4NX52TA5AO5IRJPNHFWTPG33A, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.147367857 (456h10m47.915051282s) [noise=true , address=45.91.116.13:4160 , piece_id=EWF3LPPZIRDMYSZLNEB6NEJ2DPOEBODCGHUIGI3VSVDZONIGHTTA, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.150091364 (456h10m47.917774794s) [noise=true , address=45.91.116.12:4248 , piece_id=PWKHM3QF5NV73E2BGPAESXGQL7EMCJX6PTL3226KPXXTY2HFCJMA, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:23.693098697 (456h10m48.460782123s) [noise=true , address=64.49.13.137:4495 , piece_id=5BID2HXBN532CUD3CGMVCDKLRK5PZ2JM6X2FEZEGFYDEJBZEWKZQ, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:57:24.963770925 (456h10m49.73145435s) [noise=true , address=64.49.13.169:4711 , piece_id=DK7A6JOWXKKLVH2ARPLKV5MJGIJ2KRLUPV3CKXC45BKOJATREX5Q, seg_pos={0 0}, upload=474774]: Done uploading piece. err: <nil> Nov 30 09:58:22.988765388 (456h11m47.756448819s) [noise=true , address=77.90.31.18:4813 , piece_id=WFWS6S5NX2FS63VFLXEIGQ24X4TJU3KPPV3LGJHILZWB6EBC4RUQ, seg_pos={0 0}, upload=474774]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: write tcp 159.223.143.39:44078->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:22.988845655 (456h11m47.756529087s) [noise=true , address=77.90.31.11:4569 , piece_id=BICAVERJ7E3KIGZDEQPGN3DJXHI4KC3CKBJX35HKKSUTOXDLW6TA, seg_pos={0 0}, upload=474774]: Done uploading piece. err: ecclient: upload failed (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ, address:77.90.31.11:4569): protocol: rpc: dial tcp 77.90.31.11:4569: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:22.989055796 (456h11m47.756739224s) [noise=true , address=77.90.31.13:4659 , piece_id=TJGNIW2P3ZA2RMWBN46Q5CKHNO6UCLXYRFDMI6FT6LFRNHF2ZIEQ, seg_pos={0 0}, upload=474774]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: rpc: dial tcp 77.90.31.13:4659: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:22.989256824 (456h11m47.756940255s) [noise=true , address=77.90.31.15:4725 , piece_id=5ZCXRVFD3FHULPIH3VRZ4Y2FWNLN3FWM57RQBID77Q2GCPJYVX2Q, seg_pos={0 0}, upload=474774]: Done uploading piece. err: ecclient: upload failed (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, address:77.90.31.15:4725): protocol: rpc: dial tcp 77.90.31.15:4725: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:22.989328667 (456h11m47.757012101s) [noise=true , address=77.90.31.10:4524 , piece_id=573X3XDKEJGSWY6LD7NRX2UCI4KRXEHQ3JVNU6I2OXE4N2TBTM2A, seg_pos={0 0}, upload=474774]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:53758->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:22.989628300 (456h11m47.757311732s) [noise=true , address=77.90.31.19:4872 , piece_id=5GYPMXHMRS3F263YJNUEFBSFSUVLXRAQBELHW7QQYNUNOVRM5DEQ, seg_pos={0 0}, upload=474774]: Done uploading piece. err: ecclient: upload failed (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL, address:77.90.31.19:4872): protocol: write tcp 159.223.143.39:49330->77.90.31.19:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:22.989799109 (456h11m47.757482538s) [noise=true , address=77.90.31.21:4893 , piece_id=EB4RNEEBH5S72XKGQNNFPWSTE5UKQBNXQLCKMTFFVIHC6AIP6GSQ, seg_pos={0 0}, upload=474774]: Done uploading piece. err: ecclient: upload failed (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, address:77.90.31.21:4893): protocol: write tcp 159.223.143.39:46812->77.90.31.21:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:22.989833971 (456h11m47.757517385s) [seg_pos={0 0}, upload=474774]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 09:58:23.013588934 (456h11m47.781272362s) [upload=474774]: Done uploading segments... Nov 30 09:58:23.013624374 (456h11m47.781307803s) [upload=474774]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.027004911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.021589174s trace: https://jaeger.dev.storj.io/trace/123dfebda3ba81a8 FAIL (2025-11-30 09:58:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Nov 30 09:57:19.218715858 (456h10m43.986399284s) [upload=474771]: Starting upload Nov 30 09:57:19.218774265 (456h10m43.986457693s) [upload=474771]: Done starting upload Nov 30 09:57:19.218973521 (456h10m43.986656952s) [upload=474771]: Uploading segments... Nov 30 09:57:19.219265253 (456h10m43.986948683s) [upload=474771]: Got next segment. Inline: false Nov 30 09:57:19.219340028 (456h10m43.987023456s) [upload=474771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 09:57:19.219346405 (456h10m43.98702983s) [upload=474771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 09:57:19.248628201 (456h10m44.016311626s) [seg_pos={0 0}, upload=474771]: Begin upload segment... Nov 30 09:57:19.248678963 (456h10m44.016362392s) [seg_pos={0 0}, upload=474771]: Done begin upload segment. Nov 30 09:57:19.248717029 (456h10m44.016400463s) [noise=true , address=64.49.13.26:4955 , piece_id=Y6HNUJBHC276B3G7GMZDSPHHY6JYFVA6DSAJPWQERMTL222NY77Q, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.248772084 (456h10m44.016455509s) [noise=true , address=45.91.116.12:4246 , piece_id=KXRVRA2DMEQIPBZFL4KGM7TKK4DTZS6GZCLSWD3BJWLHDNTUP7PQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.249356565 (456h10m44.017039999s) [noise=true , address=77.90.31.11:4576 , piece_id=HXHMYEKDQXYCH4P5HQUXBH6LHEZTZXOCXFAOL32IKSOBXM7HATVQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.249913382 (456h10m44.017596817s) [noise=true , address=145.79.227.39:4046 , piece_id=6SKMWRCUWJNZHCIDBTCTIGEAVNXGH6MT2FZAYSRN4XRHVVPHXFPA, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.250476525 (456h10m44.01815996s) [noise=true , address=64.49.13.140:4511 , piece_id=WIS2RRXVTQ5FH7BIYWD4WNQ4LZYYE3WPXM3QICN4OTKFIIAIYJTQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.250683679 (456h10m44.018367108s) [noise=true , address=77.90.31.12:4595 , piece_id=JWXLLLC2L6MRQDMYT5P53Q6RZTFZZIIC7IVZR74MVMEQYAEGKLEQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.251168237 (456h10m44.018851672s) [noise=true , address=64.49.13.139:4474 , piece_id=3453XOU6CABM54LMOO52J7PUFDI4MSY6GSB7FW6AYUZMEGII2FRQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.251683116 (456h10m44.019366541s) [noise=true , address=45.91.116.10:4104 , piece_id=UOKEBOB26BZM7N35CNFAMHXHX4M6UA3XO34DQ3YSWP6S7RXLJZQQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.252167206 (456h10m44.019850642s) [noise=true , address=45.91.116.13:4171 , piece_id=2CKLGYQ4O4RXS7SZP757NN7TGF3NO3RW2YVGOHSQNKBFTWYY7IRA, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.252674186 (456h10m44.020357619s) [noise=true , address=77.90.31.16:4743 , piece_id=VRVMKNY6M534FXF5XXRAMQGSZM2S6AJP76AUPEMTRAV3LVULTGMQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.253159385 (456h10m44.020842802s) [noise=true , address=145.79.227.14:4842 , piece_id=RM3WCZLCTOXF4UB3HOJYMZU6CR2MWEQJMWLHR6WXWDXHZJOUABQQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.254502627 (456h10m44.022186054s) [noise=true , address=45.91.116.8:4053 , piece_id=K6CHKGRXFHEHW4ZIDCYOQNHNMIFT5FECOAF3RFZ4LFFPVEXF77BQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.255675487 (456h10m44.023358921s) [noise=true , address=77.90.31.21:4910 , piece_id=LFVMA5ORSG64LWJZLQ4QBEWG6IDVFDRM7GGCXB6LPP3A3P6MTTEQ, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.256859564 (456h10m44.024543001s) [noise=true , address=145.79.227.2:4135 , piece_id=IWFK3S4MQAZOXZGO2N2CAFPHUAYYIGWIKJOAD7MX6Q6ACJW6B2JA, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.259287710 (456h10m44.02697117s) [noise=true , address=45.91.116.6:4000 , piece_id=LYCXX3YZFZPDDBWAZKF6ACRGFJAMLNHOSLPHT54J4PNHXLQ5WIFA, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.260997034 (456h10m44.028680463s) [noise=true , address=77.90.31.17:4791 , piece_id=HUQSVT5MD5PK6JTMIZ3FYKUT7DNTBSCHQ3JP43YHJMN3TWS3XBDA, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.261824856 (456h10m44.029508288s) [noise=true , address=45.91.116.7:4020 , piece_id=YHIAELXVIJUNILFIBBC72DVSFJDRYSQK65S2ZMZRRNBRFQPSMZNA, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.263193247 (456h10m44.03087668s) [noise=true , address=64.49.13.137:4499 , piece_id=EZZAOCAJYXFV4L6I54QK5F7SBTUTQGALQXWLSQATLMKJYKIWA75A, seg_pos={0 0}, upload=474771]: Uploading piece... Nov 30 09:57:19.328939905 (456h10m44.096623335s) [upload=474771]: Next returned nil segment Nov 30 09:57:19.328955855 (456h10m44.096639288s) [upload=474771]: Waiting for error group managing segments... Nov 30 09:57:19.333732168 (456h10m44.101415596s) [noise=true , address=45.91.116.6:4000 , piece_id=LYCXX3YZFZPDDBWAZKF6ACRGFJAMLNHOSLPHT54J4PNHXLQ5WIFA, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.333846198 (456h10m44.101529629s) [noise=true , address=45.91.116.10:4104 , piece_id=UOKEBOB26BZM7N35CNFAMHXHX4M6UA3XO34DQ3YSWP6S7RXLJZQQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.334456233 (456h10m44.102139659s) [noise=true , address=45.91.116.13:4171 , piece_id=2CKLGYQ4O4RXS7SZP757NN7TGF3NO3RW2YVGOHSQNKBFTWYY7IRA, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.336204002 (456h10m44.103887427s) [noise=true , address=145.79.227.2:4135 , piece_id=IWFK3S4MQAZOXZGO2N2CAFPHUAYYIGWIKJOAD7MX6Q6ACJW6B2JA, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.336944102 (456h10m44.104627527s) [noise=true , address=145.79.227.39:4046 , piece_id=6SKMWRCUWJNZHCIDBTCTIGEAVNXGH6MT2FZAYSRN4XRHVVPHXFPA, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.336981488 (456h10m44.104664932s) [noise=true , address=145.79.227.14:4842 , piece_id=RM3WCZLCTOXF4UB3HOJYMZU6CR2MWEQJMWLHR6WXWDXHZJOUABQQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.340148140 (456h10m44.107831577s) [noise=true , address=45.91.116.12:4246 , piece_id=KXRVRA2DMEQIPBZFL4KGM7TKK4DTZS6GZCLSWD3BJWLHDNTUP7PQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.352935532 (456h10m44.12061896s) [noise=true , address=45.91.116.7:4020 , piece_id=YHIAELXVIJUNILFIBBC72DVSFJDRYSQK65S2ZMZRRNBRFQPSMZNA, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.354412776 (456h10m44.122096201s) [noise=true , address=45.91.116.8:4053 , piece_id=K6CHKGRXFHEHW4ZIDCYOQNHNMIFT5FECOAF3RFZ4LFFPVEXF77BQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.861669676 (456h10m44.6293531s) [noise=true , address=64.49.13.26:4955 , piece_id=Y6HNUJBHC276B3G7GMZDSPHHY6JYFVA6DSAJPWQERMTL222NY77Q, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.885152939 (456h10m44.652836366s) [noise=true , address=64.49.13.139:4474 , piece_id=3453XOU6CABM54LMOO52J7PUFDI4MSY6GSB7FW6AYUZMEGII2FRQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:19.942890026 (456h10m44.71057345s) [noise=true , address=64.49.13.140:4511 , piece_id=WIS2RRXVTQ5FH7BIYWD4WNQ4LZYYE3WPXM3QICN4OTKFIIAIYJTQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:57:20.065003449 (456h10m44.832686877s) [noise=true , address=64.49.13.137:4499 , piece_id=EZZAOCAJYXFV4L6I54QK5F7SBTUTQGALQXWLSQATLMKJYKIWA75A, seg_pos={0 0}, upload=474771]: Done uploading piece. err: <nil> Nov 30 09:58:19.179496049 (456h11m43.947179474s) [noise=true , address=77.90.31.16:4743 , piece_id=VRVMKNY6M534FXF5XXRAMQGSZM2S6AJP76AUPEMTRAV3LVULTGMQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: write tcp 159.223.143.39:60832->77.90.31.16:4743: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:19.179536837 (456h11m43.947220265s) [noise=true , address=77.90.31.17:4791 , piece_id=HUQSVT5MD5PK6JTMIZ3FYKUT7DNTBSCHQ3JP43YHJMN3TWS3XBDA, seg_pos={0 0}, upload=474771]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: rpc: dial tcp 77.90.31.17:4791: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:19.179665408 (456h11m43.947348837s) [noise=true , address=77.90.31.21:4910 , piece_id=LFVMA5ORSG64LWJZLQ4QBEWG6IDVFDRM7GGCXB6LPP3A3P6MTTEQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: write tcp 159.223.143.39:35362->77.90.31.21:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:19.179754975 (456h11m43.9474384s) [noise=true , address=77.90.31.12:4595 , piece_id=JWXLLLC2L6MRQDMYT5P53Q6RZTFZZIIC7IVZR74MVMEQYAEGKLEQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 159.223.143.39:57212->77.90.31.12:4595: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:19.179826911 (456h11m43.94751034s) [noise=true , address=77.90.31.11:4576 , piece_id=HXHMYEKDQXYCH4P5HQUXBH6LHEZTZXOCXFAOL32IKSOBXM7HATVQ, seg_pos={0 0}, upload=474771]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: write tcp 159.223.143.39:56046->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:19.179852180 (456h11m43.947535599s) [seg_pos={0 0}, upload=474771]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 30 09:58:19.199194244 (456h11m43.96687767s) [upload=474771]: Done uploading segments... Nov 30 09:58:19.199228058 (456h11m43.966911486s) [upload=474771]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.021618779s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.002163938s trace: https://jaeger.dev.storj.io/trace/664c6bfe4fc906ff FAIL (2025-11-30 09:57:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002192716s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 42.11471ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:35:22 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 42.140903ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 38.839549ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:34:58 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 38.867097ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 42.083852ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:34:42 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 42.109091ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 37.767078ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:33:35 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 37.789163ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 45.661956ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:32:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 45.703387ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 42.737242ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:31:43 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 42.768292ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 41.653833ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:30:56 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 41.682185ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 40.589456ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:29:59 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 40.613107ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 44.075999ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:29:17 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 44.099006ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 41.274027ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:28:09 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 41.302009ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 39.711326ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:27:50 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 39.741095ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 42.537671ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/382204e96ab604c6 FAIL (2025-11-30 03:26:57 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 42.588855ms edge gateway vhost us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)