Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [............XXXXXXX.........................................] duration:
425.54371ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:32:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 425.56627ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration:
398.643614ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:32:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 398.668554ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [................XXXXX.......................................] duration:
528.820264ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:32:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 528.848173ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............XXXXX..........................................] duration:
469.158012ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:32:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 469.185084ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.............XXXXXXX........................................] duration:
435.465315ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:31:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 435.504705ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.................XXXXX......................................] duration:
626.019102ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:31:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 626.043762ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration:
417.886879ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:31:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 417.911261ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............XXXXX.........................................] duration:
410.888694ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:31:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 410.913095ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration:
412.099255ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:30:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 412.12251ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..............XXXXXXX.......................................] duration:
417.317165ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:30:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 417.338825ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............XXXXX........................................] duration:
317.212885ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:30:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 317.241918ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration:
508.868964ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:30:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 508.894398ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration:
433.981477ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:30:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 434.01096ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...............XXXXXXX......................................] duration:
417.484993ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:29:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 417.505648ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............XXXXX..........................................] duration:
383.947611ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:29:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 383.976519ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration:
488.900721ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:29:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 488.921998ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [................XXXXX.......................................] duration:
392.005006ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:29:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 392.029476ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [................XXXXXXX.....................................] duration:
381.291738ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:29:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 381.322513ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................XXXXX......................................] duration:
347.87315ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:28:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 347.908302ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration:
502.814348ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:28:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.843551ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.................XXXXXXX....................................] duration:
429.755972ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:28:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 429.780508ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............XXXXX.........................................] duration:
357.553955ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:28:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 357.582363ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration:
358.983467ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:28:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 359.011191ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration:
581.154804ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:28:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 581.200232ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration:
867.360653ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:27:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 867.383547ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............XXXXX........................................] duration:
402.470956ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:27:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 402.498623ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..................XXXXXXX...................................] duration:
381.147147ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:27:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 381.171132ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration:
498.672002ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:27:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 498.696201ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...................XXXXXXX..................................] duration:
385.476491ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:27:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 385.502876ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [................XXXXX.......................................] duration:
401.125347ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:26:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 401.153719ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration:
410.111614ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:26:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 410.137901ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......................XXXXX................................] duration:
541.574431ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:26:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 541.598396ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration:
591.052026ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:26:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 591.086156ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration:
336.84803ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:26:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 336.87173ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................XXXXX......................................] duration:
381.019818ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:26:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 381.042813ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [....................XXXXXXX.................................] duration:
390.115808ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:26:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 390.162851ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration:
362.702399ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:26:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 362.728124ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration:
534.879931ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:26:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 534.945295ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration:
342.239267ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:25:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 342.269829ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.....................XXXXXXX................................] duration:
404.120207ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:25:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 404.220645ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................XXXXX................................] duration:
392.818834ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:25:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 392.845654ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration:
472.402867ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:25:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.438376ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [......................XXXXXXX...............................] duration:
386.00903ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:24:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 386.037101ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration:
373.19377ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:24:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 373.219031ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration:
440.083036ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:24:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 440.105513ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration:
628.731238ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:24:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 628.757348ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration:
367.907459ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:24:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 367.934551ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration:
414.078006ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:24:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 414.104581ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.......................XXXXXXX..............................] duration:
407.065435ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:23:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 407.085958ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [............................XXXXX...........................] duration:
542.944812ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:23:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 542.971324ms
core
uplink
us-select-1
us1-placement-20
|