Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................XX.......................................] duration: 484.82097ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:02:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.844751ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 562.802728ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:02:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 562.84181ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 410.297797ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:02:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 410.397188ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 432.796125ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:01:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 432.822433ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 402.71994ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:01:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 402.754626ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 497.515108ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:01:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 497.639233ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................XX......................................] duration: 439.194402ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:01:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 439.223281ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 421.214633ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:01:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 421.244628ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 466.540784ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:00:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 466.5718ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 501.210469ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:00:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 501.285114ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................XX.....................................] duration: 426.466802ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:00:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.505634ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 503.957909ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:00:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 504.059974ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 556.390965ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:59:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 556.421001ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................XX....................................] duration: 426.123303ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:59:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.148223ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 549.501629ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:59:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 549.530267ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 388.01307ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:59:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 388.233458ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 368.017257ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:59:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 368.040016ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 532.099765ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:59:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 532.152794ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................XX...................................] duration: 604.982384ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:59:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.007121ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................XX..................................] duration: 451.520376ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:58:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.54369ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 572.612668ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:58:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 572.636319ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 553.678715ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:58:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 553.707293ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................XX.................................] duration: 423.591765ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:58:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.616622ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 388.304951ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:58:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 388.339865ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 398.861304ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:57:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 398.894129ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 520.421529ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:57:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 520.528202ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...........................X...............................] duration: 451.542204ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:57:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 451.581938ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................XX................................] duration: 432.498775ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:57:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 432.587673ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 601.646903ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:56:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 601.673022ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 358.423202ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:56:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 358.448817ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...........................X..............................] duration: 486.941656ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:56:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 487.034813ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................XX...............................] duration: 440.182043ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:55:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 440.272079ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 390.433563ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:55:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 390.525918ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...........................X.............................] duration: 454.135516ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:55:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 454.186791ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 468.941659ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:55:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 469.073512ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................XX..............................] duration: 446.217442ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:54:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.242315ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 472.118049ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:54:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 472.142946ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...........................X............................] duration: 502.508755ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:54:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 502.534043ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 550.415723ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:54:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 550.439201ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................XX.............................] duration: 437.449401ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:54:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 437.471003ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...........................X...........................] duration: 456.467725ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:53:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 456.512157ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.............................XX............................] duration: 469.803115ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:53:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.824972ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 381.89667ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:53:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 381.911401ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 488.184042ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:53:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 488.218993ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XX.............................XX...........................] duration: 447.311651ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:53:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 447.331688ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...............................X...........................] duration: 404.167499ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:52:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 404.197203ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........................X..........................] duration: 467.531512ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:52:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 467.58391ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..............................X............................] duration: 537.409832ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:52:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 537.437426ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.XX.............................XX..........................] duration: 459.560493ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:52:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 459.624846ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..............................X...........................] duration: 481.634215ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:52:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 481.739781ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)