Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............XX....X......XX.XXXX...X......................] duration: 25.113119276s FAIL (2025-07-27 00:55:56 UTC) orbiter warning: context deadline exceeded Duration: 25.113175288s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......................X......X.......] duration: 252.805385ms FAIL (2025-07-27 00:55:43 UTC) orbiter warning: uplink: too many requests Duration: 252.846661ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X..] duration: 1.858564585s FAIL (2025-07-27 00:55:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ecestore: piecestore close: conn closed: op Jul 27 00:52:42.873478110 (3h0m58.654411477s) [noise=true , address=104.36.82.9:4005 , piece_id=3FPIL4WOQG3DOFAMQZEP3D5LUQ6NBDWXYDP4JCIPR6S6RMT7NFSQ, seg_pos={0 0}, upload=2798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: write tcp 159.223.143.39:52664->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:42.873503535 (3h0m58.654436911s) [seg_pos={0 0}, upload=2798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:42.873590035 (3h0m58.6545234s) [upload=2798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:42.873595303 (3h0m58.654528678s) [upload=2798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:42.937011195 (3h0m58.717944581s) [upload=2798]: Done uploading segments... Jul 27 00:52:42.937035418 (3h0m58.717968805s) [upload=2798]: Upload finished. err: <nil> Jul 27 00:53:55.126015126 (3h2m10.90694851s) [upload=2817]: Starting upload Jul 27 00:53:55.126135869 (3h2m10.907069254s) [upload=2817]: Done starting upload Jul 27 00:53:55.126491322 (3h2m10.907424686s) [upload=2817]: Uploading segments... Jul 27 00:53:55.126557628 (3h2m10.907491015s) [upload=2817]: Got next segment. Inline: false Jul 27 00:53:55.126599255 (3h2m10.907532631s) [upload=2817]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:53:55.126605523 (3h2m10.90753891s) [upload=2817]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:53:55.155376273 (3h2m10.936309657s) [seg_pos={0 0}, upload=2817]: Begin upload segment... Jul 27 00:53:55.155440154 (3h2m10.936373543s) [seg_pos={0 0}, upload=2817]: Done begin upload segment. Jul 27 00:53:55.155470937 (3h2m10.936404323s) [noise=true , address=77.90.31.16:4734 , piece_id=WBI2ZFHG3RA2D4WD4AJGJ5OWSOA6AN2AJC5XDC667VZT6X7NMKAQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.155494226 (3h2m10.936427612s) [noise=true , address=45.91.116.13:4155 , piece_id=XT7IFLMBWP5HVWW5465JN5IY3FOCGZEVOJZYB5GIRJZIQJTQJ6GA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156059719 (3h2m10.936993104s) [noise=true , address=45.91.116.11:4117 , piece_id=GA732CTJQY7EN7SPP3AVLQP5CFSA3BCHG5KBK5Q4GYHHJYKBX6IQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156167818 (3h2m10.937101201s) [noise=true , address=104.36.82.10:4079 , piece_id=HKZEBINY4QCFPGIB5RXYZZASW63HT3OG4FTAWV7RGHS56DHXND5Q, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156505619 (3h2m10.937439005s) [noise=true , address=77.90.31.12:4590 , piece_id=S2HZZODUACLBWYUOV5EBO6LLDCCTZMSQMWZZ53MLCYCRFTCXLCJA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156562271 (3h2m10.937495653s) [noise=true , address=45.91.116.4:4940 , piece_id=M3UBVVLFROHJPLDKB3YMPSGZD43MBQEYNEFXKUL4RX5ZJWGSZ3XQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156891279 (3h2m10.937824663s) [noise=true , address=45.91.116.5:4298 , piece_id=4EHZK7XJTXWA7BS4U3MQFVQZVIZ3OZCL6PJ47YORS4GDRDCXL4OA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156903943 (3h2m10.937837319s) [noise=true , address=45.91.116.10:4083 , piece_id=GDL2K5CRFMCCOTGJ5KR6W5JME7EGBGU6TC3BXOUQRELCK64F6HBA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157282224 (3h2m10.9382156s) [noise=true , address=104.36.82.5:4838 , piece_id=MXSG77V6UZ57P3YRD3LOIUYIWEZQH53ST4VRO6DXL4GOB5W4YSEQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157306090 (3h2m10.938239477s) [noise=true , address=104.36.82.4:4803 , piece_id=RCYJC3FODEX24UKBENVN4YPQCRVSUJSEMF2CBRITBFK5D3NYXINQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157642347 (3h2m10.938575725s) [noise=true , address=45.91.116.9:4261 , piece_id=HVOUDZVMC7INGNSJFXVJ3NUOQEJB3LNGMHCLRPY7ACC7GUWLCYQA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157707311 (3h2m10.938640697s) [noise=true , address=104.36.82.14:4223 , piece_id=RIVE3LGUDCOFZNJXJUFGNSVRMM3O7WBPJOSIEYT7T3EN3OMZAJ7Q, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158092892 (3h2m10.93902627s) [noise=true , address=77.90.31.11:4554 , piece_id=TFOXBA4DTT42BMTD63CUYHHCRDLVYCNM27VBO2AGIDV4RQTEF7UQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158535071 (3h2m10.939468457s) [noise=true , address=77.90.31.13:4628 , piece_id=DIJDNVIO4GFHX2OQSX3ID65Y7DE75CSBP2JJ23J5SCX645FSLTKA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158987393 (3h2m10.939920779s) [noise=true , address=45.91.116.12:4225 , piece_id=C6TIRTKRRXTZCV3XJLLVQ3D4USCZOOJALQZGL7TCWYUD45R7M5JQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.159442456 (3h2m10.940375843s) [noise=true , address=77.90.31.14:4661 , piece_id=JWB2QDJXKUKGFRI7JXYK7Y53INTFQZSUDM5AUA2YMNAUCZYZU2UA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.253097500 (3h2m11.034030882s) [upload=2817]: Next returned nil segment Jul 27 00:53:55.253118666 (3h2m11.034052055s) [upload=2817]: Waiting for error group managing segments... Jul 27 00:53:55.258249685 (3h2m11.03918309s) [noise=true , address=45.91.116.13:4155 , piece_id=XT7IFLMBWP5HVWW5465JN5IY3FOCGZEVOJZYB5GIRJZIQJTQJ6GA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258332604 (3h2m11.039265967s) [noise=true , address=77.90.31.13:4628 , piece_id=DIJDNVIO4GFHX2OQSX3ID65Y7DE75CSBP2JJ23J5SCX645FSLTKA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258918124 (3h2m11.0398515s) [noise=true , address=45.91.116.4:4940 , piece_id=M3UBVVLFROHJPLDKB3YMPSGZD43MBQEYNEFXKUL4RX5ZJWGSZ3XQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258975807 (3h2m11.039909184s) [noise=true , address=45.91.116.11:4117 , piece_id=GA732CTJQY7EN7SPP3AVLQP5CFSA3BCHG5KBK5Q4GYHHJYKBX6IQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.259515850 (3h2m11.04044923s) [noise=true , address=77.90.31.16:4734 , piece_id=WBI2ZFHG3RA2D4WD4AJGJ5OWSOA6AN2AJC5XDC667VZT6X7NMKAQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.259635850 (3h2m11.040569217s) [noise=true , address=45.91.116.9:4261 , piece_id=HVOUDZVMC7INGNSJFXVJ3NUOQEJB3LNGMHCLRPY7ACC7GUWLCYQA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260090390 (3h2m11.041023766s) [noise=true , address=45.91.116.5:4298 , piece_id=4EHZK7XJTXWA7BS4U3MQFVQZVIZ3OZCL6PJ47YORS4GDRDCXL4OA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260242555 (3h2m11.041175936s) [noise=true , address=77.90.31.11:4554 , piece_id=TFOXBA4DTT42BMTD63CUYHHCRDLVYCNM27VBO2AGIDV4RQTEF7UQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260723294 (3h2m11.041656687s) [noise=true , address=77.90.31.14:4661 , piece_id=JWB2QDJXKUKGFRI7JXYK7Y53INTFQZSUDM5AUA2YMNAUCZYZU2UA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260867397 (3h2m11.041800775s) [noise=true , address=45.91.116.12:4225 , piece_id=C6TIRTKRRXTZCV3XJLLVQ3D4USCZOOJALQZGL7TCWYUD45R7M5JQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.435094721 (3h2m11.216028114s) [noise=true , address=45.91.116.10:4083 , piece_id=GDL2K5CRFMCCOTGJ5KR6W5JME7EGBGU6TC3BXOUQRELCK64F6HBA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:56.616631435 (3h2m12.397564818s) [noise=true , address=77.90.31.12:4590 , piece_id=S2HZZODUACLBWYUOV5EBO6LLDCCTZMSQMWZZ53MLCYCRFTCXLCJA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:54:25.094965082 (3h2m40.875898466s) [noise=true , address=104.36.82.14:4223 , piece_id=RIVE3LGUDCOFZNJXJUFGNSVRMM3O7WBPJOSIEYT7T3EN3OMZAJ7Q, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:47682->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095080315 (3h2m40.876013703s) [noise=true , address=104.36.82.10:4079 , piece_id=HKZEBINY4QCFPGIB5RXYZZASW63HT3OG4FTAWV7RGHS56DHXND5Q, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:60438->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095113588 (3h2m40.876046958s) [noise=true , address=104.36.82.5:4838 , piece_id=MXSG77V6UZ57P3YRD3LOIUYIWEZQH53ST4VRO6DXL4GOB5W4YSEQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095322027 (3h2m40.876255413s) [noise=true , address=104.36.82.4:4803 , piece_id=RCYJC3FODEX24UKBENVN4YPQCRVSUJSEMF2CBRITBFK5D3NYXINQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:54:25.095364007 (3h2m40.876297394s) [seg_pos={0 0}, upload=2817]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:25.127534016 (3h2m40.908467396s) [upload=2817]: Done uploading segments... Jul 27 00:54:25.127566856 (3h2m40.908500238s) [upload=2817]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.858602071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXXX] duration: 1m0.032139738s FAIL (2025-07-27 00:55:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== :4251 , piece_id=SQW7S5SHTII2OUJU4I3XRVIVOMWSJ5XP4RFWUXO3TVCGYDSWAYPQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.755580071 (3h2m32.536513456s) [noise=true , address=45.91.116.11:4130 , piece_id=IQDMQE7H5YQMIT37UI2EFA5CFHEQNKZV4JS6ABWBLTM6XZNG2RWA, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.755739450 (3h2m32.536672827s) [noise=true , address=64.49.13.26:4953 , piece_id=JSJ5BRRJLAIBMZJXYVRASLJKRYMILR7RE63DJGJRPEQFKQIYLWGQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756111287 (3h2m32.537044652s) [noise=true , address=79.127.201.215:4354 , piece_id=PWBILOHFL4LFPEXJRMDD7AI7B5BGRLZDGF2I3GG5JTGHKKVTSPWQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756318393 (3h2m32.53725177s) [noise=true , address=79.127.205.245:4236 , piece_id=KK2NX2OYSBEW2ZFHYNDEAZMDLW3OH3TRZC2QLX65IU5OVGP6LCMQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756429646 (3h2m32.537363082s) [noise=true , address=64.49.13.137:4502 , piece_id=6IJQDAF32P32X4FFBRRACCUGMOZ2L3NQ43QUPBDGHEAR6RWUB3QQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756711422 (3h2m32.537644801s) [noise=true , address=64.49.13.170:4717 , piece_id=VB3GSOCJSB57DZFK33JMJSQVBOQCUIEWBQAL7OXPORQBASZFTHIQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.756756090 (3h2m32.537689469s) [noise=true , address=104.36.82.54:4192 , piece_id=7MS5L7BEZDH2QQFAGZNOLKAKFST6JMPRPRD3G65ZDSIEHP7TRKWQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.757047069 (3h2m32.537980437s) [noise=true , address=77.90.31.17:4795 , piece_id=T3ZB7QBQCF2FVIFIOCEH2JCXTGGSUZ4JLSRG5GE5QHR724C22X6A, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.757218726 (3h2m32.538152103s) [noise=true , address=79.127.205.235:4692 , piece_id=MCGCVQWX5SF2JRLLT5VJCJHIO6ODTO3DWGVGDIOY67ANCU5SWA2Q, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.757469923 (3h2m32.538403308s) [noise=true , address=104.36.82.46:4489 , piece_id=SXRMCGBEI6AHONXQNBOZWVIJUTKVHNIMBRYUOQETOZOEQA7FLL5Q, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.757864967 (3h2m32.538798346s) [noise=true , address=104.36.82.56:4229 , piece_id=W3KURAVN5NQWYJFEYS2RG7UNDIICXEVSDJAP3QVYAI5WX42Y73HA, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.758272858 (3h2m32.539206247s) [noise=true , address=104.36.82.52:4100 , piece_id=A5OHEXUKBOATOOPH6SJN4KLUPOTGHU7YKPKALGRG52XWGG26KUUQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.758656224 (3h2m32.539589606s) [noise=true , address=104.36.82.47:4525 , piece_id=ZGBE436YBZRW2Q6GUCZNMOWKC72Q246R74ZDJXGMNC4TD3HH5OYA, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.759053323 (3h2m32.53998672s) [noise=true , address=104.36.82.55:4206 , piece_id=K55J2F4REVE7RUJNSGN5HSJJ2OQSFZCUTNE26DM3XT76OBLUYLZQ, seg_pos={0 0}, upload=2821]: Uploading piece... Jul 27 00:54:16.993086636 (3h2m32.774020019s) [upload=2821]: Next returned nil segment Jul 27 00:54:16.993110617 (3h2m32.774044005s) [upload=2821]: Waiting for error group managing segments... Jul 27 00:54:17.374439326 (3h2m33.155372692s) [noise=true , address=79.127.205.235:4692 , piece_id=MCGCVQWX5SF2JRLLT5VJCJHIO6ODTO3DWGVGDIOY67ANCU5SWA2Q, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:17.434908092 (3h2m33.21584146s) [noise=true , address=64.49.13.170:4717 , piece_id=VB3GSOCJSB57DZFK33JMJSQVBOQCUIEWBQAL7OXPORQBASZFTHIQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:17.541315587 (3h2m33.322248967s) [noise=true , address=79.127.205.233:4650 , piece_id=BZHEPPYOKFP2IIAACX3BAR6VRU5CEOLHIFFJ5YQN3PAI5CGT3LVQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:17.787531794 (3h2m33.568465159s) [noise=true , address=64.49.13.137:4502 , piece_id=6IJQDAF32P32X4FFBRRACCUGMOZ2L3NQ43QUPBDGHEAR6RWUB3QQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 27 00:54:18.011681960 (3h2m33.792615325s) [noise=true , address=79.127.205.245:4236 , piece_id=KK2NX2OYSBEW2ZFHYNDEAZMDLW3OH3TRZC2QLX65IU5OVGP6LCMQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:18.306391519 (3h2m34.087324911s) [noise=true , address=64.49.13.26:4953 , piece_id=JSJ5BRRJLAIBMZJXYVRASLJKRYMILR7RE63DJGJRPEQFKQIYLWGQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:54:20.569130201 (3h2m36.350063586s) [noise=true , address=45.91.116.9:4277 , piece_id=BFHXR3GRRGXFLUDGUHT4SPQDJVYV7BTLTARO3ZI2BD6VB2DLHWWA, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:54:21.251259175 (3h2m37.03219255s) [noise=true , address=45.91.116.11:4130 , piece_id=IQDMQE7H5YQMIT37UI2EFA5CFHEQNKZV4JS6ABWBLTM6XZNG2RWA, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:54:21.924835570 (3h2m37.705768949s) [noise=true , address=79.127.201.215:4354 , piece_id=PWBILOHFL4LFPEXJRMDD7AI7B5BGRLZDGF2I3GG5JTGHKKVTSPWQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: <nil> Jul 27 00:55:16.687076820 (3h3m32.468010212s) [noise=true , address=104.36.82.53:4144 , piece_id=H6GASXUFBNOTUR5GTZ4FC25UNUQP4QHHOTE2ETDTKOFX5HSPCJPQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:1FfzcPNLmNdrvoZBKcmCLz8d9rYSAnjFTo4NBnxBzN6dVcTNwt, address:104.36.82.53:4144): protocol: rpc: dial tcp 104.36.82.53:4144: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687184443 (3h3m32.468117825s) [noise=true , address=104.36.82.46:4489 , piece_id=SXRMCGBEI6AHONXQNBOZWVIJUTKVHNIMBRYUOQETOZOEQA7FLL5Q, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: rpc: dial tcp 104.36.82.46:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687248550 (3h3m32.468181926s) [noise=true , address=104.36.82.14:4251 , piece_id=SQW7S5SHTII2OUJU4I3XRVIVOMWSJ5XP4RFWUXO3TVCGYDSWAYPQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: write tcp 159.223.143.39:52606->104.36.82.14:4251: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687508948 (3h3m32.468442315s) [noise=true , address=104.36.82.47:4525 , piece_id=ZGBE436YBZRW2Q6GUCZNMOWKC72Q246R74ZDJXGMNC4TD3HH5OYA, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: write tcp 159.223.143.39:44592->104.36.82.47:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687531296 (3h3m32.468464671s) [noise=true , address=104.36.82.56:4229 , piece_id=W3KURAVN5NQWYJFEYS2RG7UNDIICXEVSDJAP3QVYAI5WX42Y73HA, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: rpc: dial tcp 104.36.82.56:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687591395 (3h3m32.468524769s) [noise=true , address=104.36.82.52:4100 , piece_id=A5OHEXUKBOATOOPH6SJN4KLUPOTGHU7YKPKALGRG52XWGG26KUUQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:104.36.82.52:4100): protocol: rpc: dial tcp 104.36.82.52:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687597018 (3h3m32.468530386s) [noise=true , address=104.36.82.55:4206 , piece_id=K55J2F4REVE7RUJNSGN5HSJJ2OQSFZCUTNE26DM3XT76OBLUYLZQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:104.36.82.55:4206): protocol: rpc: dial tcp 104.36.82.55:4206: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687643383 (3h3m32.468576749s) [noise=true , address=77.90.31.17:4795 , piece_id=T3ZB7QBQCF2FVIFIOCEH2JCXTGGSUZ4JLSRG5GE5QHR724C22X6A, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:49398->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687675572 (3h3m32.468608937s) [noise=true , address=104.36.82.54:4192 , piece_id=7MS5L7BEZDH2QQFAGZNOLKAKFST6JMPRPRD3G65ZDSIEHP7TRKWQ, seg_pos={0 0}, upload=2821]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: rpc: dial tcp 104.36.82.54:4192: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:16.687745470 (3h3m32.46867885s) [seg_pos={0 0}, upload=2821]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 27 00:55:16.716410196 (3h3m32.497343608s) [upload=2821]: Done uploading segments... Jul 27 00:55:16.716442615 (3h3m32.497376s) [upload=2821]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.032161948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.....] duration: 505.628511ms FAIL (2025-07-27 00:55:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 505.661512ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......................X......X......] duration: 211.104036ms FAIL (2025-07-27 00:54:45 UTC) orbiter warning: uplink: too many requests Duration: 211.143248ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...............XX....X......XX.XXXX...X.....................] duration: 503.302404ms FAIL (2025-07-27 00:54:40 UTC) orbiter warning: context deadline exceeded Duration: 503.337037ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................X...............X...X...............X....] duration: 32.430291673s FAIL (2025-07-27 00:54:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.430886695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.X.] duration: 424.226169ms FAIL (2025-07-27 00:54:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ecestore: piecestore close: conn closed: op Jul 27 00:52:42.873478110 (3h0m58.654411477s) [noise=true , address=104.36.82.9:4005 , piece_id=3FPIL4WOQG3DOFAMQZEP3D5LUQ6NBDWXYDP4JCIPR6S6RMT7NFSQ, seg_pos={0 0}, upload=2798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: write tcp 159.223.143.39:52664->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:42.873503535 (3h0m58.654436911s) [seg_pos={0 0}, upload=2798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:42.873590035 (3h0m58.6545234s) [upload=2798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:42.873595303 (3h0m58.654528678s) [upload=2798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:42.937011195 (3h0m58.717944581s) [upload=2798]: Done uploading segments... Jul 27 00:52:42.937035418 (3h0m58.717968805s) [upload=2798]: Upload finished. err: <nil> Jul 27 00:53:55.126015126 (3h2m10.90694851s) [upload=2817]: Starting upload Jul 27 00:53:55.126135869 (3h2m10.907069254s) [upload=2817]: Done starting upload Jul 27 00:53:55.126491322 (3h2m10.907424686s) [upload=2817]: Uploading segments... Jul 27 00:53:55.126557628 (3h2m10.907491015s) [upload=2817]: Got next segment. Inline: false Jul 27 00:53:55.126599255 (3h2m10.907532631s) [upload=2817]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:53:55.126605523 (3h2m10.90753891s) [upload=2817]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:53:55.155376273 (3h2m10.936309657s) [seg_pos={0 0}, upload=2817]: Begin upload segment... Jul 27 00:53:55.155440154 (3h2m10.936373543s) [seg_pos={0 0}, upload=2817]: Done begin upload segment. Jul 27 00:53:55.155470937 (3h2m10.936404323s) [noise=true , address=77.90.31.16:4734 , piece_id=WBI2ZFHG3RA2D4WD4AJGJ5OWSOA6AN2AJC5XDC667VZT6X7NMKAQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.155494226 (3h2m10.936427612s) [noise=true , address=45.91.116.13:4155 , piece_id=XT7IFLMBWP5HVWW5465JN5IY3FOCGZEVOJZYB5GIRJZIQJTQJ6GA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156059719 (3h2m10.936993104s) [noise=true , address=45.91.116.11:4117 , piece_id=GA732CTJQY7EN7SPP3AVLQP5CFSA3BCHG5KBK5Q4GYHHJYKBX6IQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156167818 (3h2m10.937101201s) [noise=true , address=104.36.82.10:4079 , piece_id=HKZEBINY4QCFPGIB5RXYZZASW63HT3OG4FTAWV7RGHS56DHXND5Q, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156505619 (3h2m10.937439005s) [noise=true , address=77.90.31.12:4590 , piece_id=S2HZZODUACLBWYUOV5EBO6LLDCCTZMSQMWZZ53MLCYCRFTCXLCJA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156562271 (3h2m10.937495653s) [noise=true , address=45.91.116.4:4940 , piece_id=M3UBVVLFROHJPLDKB3YMPSGZD43MBQEYNEFXKUL4RX5ZJWGSZ3XQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156891279 (3h2m10.937824663s) [noise=true , address=45.91.116.5:4298 , piece_id=4EHZK7XJTXWA7BS4U3MQFVQZVIZ3OZCL6PJ47YORS4GDRDCXL4OA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.156903943 (3h2m10.937837319s) [noise=true , address=45.91.116.10:4083 , piece_id=GDL2K5CRFMCCOTGJ5KR6W5JME7EGBGU6TC3BXOUQRELCK64F6HBA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157282224 (3h2m10.9382156s) [noise=true , address=104.36.82.5:4838 , piece_id=MXSG77V6UZ57P3YRD3LOIUYIWEZQH53ST4VRO6DXL4GOB5W4YSEQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157306090 (3h2m10.938239477s) [noise=true , address=104.36.82.4:4803 , piece_id=RCYJC3FODEX24UKBENVN4YPQCRVSUJSEMF2CBRITBFK5D3NYXINQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157642347 (3h2m10.938575725s) [noise=true , address=45.91.116.9:4261 , piece_id=HVOUDZVMC7INGNSJFXVJ3NUOQEJB3LNGMHCLRPY7ACC7GUWLCYQA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.157707311 (3h2m10.938640697s) [noise=true , address=104.36.82.14:4223 , piece_id=RIVE3LGUDCOFZNJXJUFGNSVRMM3O7WBPJOSIEYT7T3EN3OMZAJ7Q, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158092892 (3h2m10.93902627s) [noise=true , address=77.90.31.11:4554 , piece_id=TFOXBA4DTT42BMTD63CUYHHCRDLVYCNM27VBO2AGIDV4RQTEF7UQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158535071 (3h2m10.939468457s) [noise=true , address=77.90.31.13:4628 , piece_id=DIJDNVIO4GFHX2OQSX3ID65Y7DE75CSBP2JJ23J5SCX645FSLTKA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.158987393 (3h2m10.939920779s) [noise=true , address=45.91.116.12:4225 , piece_id=C6TIRTKRRXTZCV3XJLLVQ3D4USCZOOJALQZGL7TCWYUD45R7M5JQ, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.159442456 (3h2m10.940375843s) [noise=true , address=77.90.31.14:4661 , piece_id=JWB2QDJXKUKGFRI7JXYK7Y53INTFQZSUDM5AUA2YMNAUCZYZU2UA, seg_pos={0 0}, upload=2817]: Uploading piece... Jul 27 00:53:55.253097500 (3h2m11.034030882s) [upload=2817]: Next returned nil segment Jul 27 00:53:55.253118666 (3h2m11.034052055s) [upload=2817]: Waiting for error group managing segments... Jul 27 00:53:55.258249685 (3h2m11.03918309s) [noise=true , address=45.91.116.13:4155 , piece_id=XT7IFLMBWP5HVWW5465JN5IY3FOCGZEVOJZYB5GIRJZIQJTQJ6GA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258332604 (3h2m11.039265967s) [noise=true , address=77.90.31.13:4628 , piece_id=DIJDNVIO4GFHX2OQSX3ID65Y7DE75CSBP2JJ23J5SCX645FSLTKA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258918124 (3h2m11.0398515s) [noise=true , address=45.91.116.4:4940 , piece_id=M3UBVVLFROHJPLDKB3YMPSGZD43MBQEYNEFXKUL4RX5ZJWGSZ3XQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.258975807 (3h2m11.039909184s) [noise=true , address=45.91.116.11:4117 , piece_id=GA732CTJQY7EN7SPP3AVLQP5CFSA3BCHG5KBK5Q4GYHHJYKBX6IQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.259515850 (3h2m11.04044923s) [noise=true , address=77.90.31.16:4734 , piece_id=WBI2ZFHG3RA2D4WD4AJGJ5OWSOA6AN2AJC5XDC667VZT6X7NMKAQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.259635850 (3h2m11.040569217s) [noise=true , address=45.91.116.9:4261 , piece_id=HVOUDZVMC7INGNSJFXVJ3NUOQEJB3LNGMHCLRPY7ACC7GUWLCYQA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260090390 (3h2m11.041023766s) [noise=true , address=45.91.116.5:4298 , piece_id=4EHZK7XJTXWA7BS4U3MQFVQZVIZ3OZCL6PJ47YORS4GDRDCXL4OA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260242555 (3h2m11.041175936s) [noise=true , address=77.90.31.11:4554 , piece_id=TFOXBA4DTT42BMTD63CUYHHCRDLVYCNM27VBO2AGIDV4RQTEF7UQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260723294 (3h2m11.041656687s) [noise=true , address=77.90.31.14:4661 , piece_id=JWB2QDJXKUKGFRI7JXYK7Y53INTFQZSUDM5AUA2YMNAUCZYZU2UA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.260867397 (3h2m11.041800775s) [noise=true , address=45.91.116.12:4225 , piece_id=C6TIRTKRRXTZCV3XJLLVQ3D4USCZOOJALQZGL7TCWYUD45R7M5JQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:55.435094721 (3h2m11.216028114s) [noise=true , address=45.91.116.10:4083 , piece_id=GDL2K5CRFMCCOTGJ5KR6W5JME7EGBGU6TC3BXOUQRELCK64F6HBA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:53:56.616631435 (3h2m12.397564818s) [noise=true , address=77.90.31.12:4590 , piece_id=S2HZZODUACLBWYUOV5EBO6LLDCCTZMSQMWZZ53MLCYCRFTCXLCJA, seg_pos={0 0}, upload=2817]: Done uploading piece. err: <nil> Jul 27 00:54:25.094965082 (3h2m40.875898466s) [noise=true , address=104.36.82.14:4223 , piece_id=RIVE3LGUDCOFZNJXJUFGNSVRMM3O7WBPJOSIEYT7T3EN3OMZAJ7Q, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:47682->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095080315 (3h2m40.876013703s) [noise=true , address=104.36.82.10:4079 , piece_id=HKZEBINY4QCFPGIB5RXYZZASW63HT3OG4FTAWV7RGHS56DHXND5Q, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:60438->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095113588 (3h2m40.876046958s) [noise=true , address=104.36.82.5:4838 , piece_id=MXSG77V6UZ57P3YRD3LOIUYIWEZQH53ST4VRO6DXL4GOB5W4YSEQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:25.095322027 (3h2m40.876255413s) [noise=true , address=104.36.82.4:4803 , piece_id=RCYJC3FODEX24UKBENVN4YPQCRVSUJSEMF2CBRITBFK5D3NYXINQ, seg_pos={0 0}, upload=2817]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:54:25.095364007 (3h2m40.876297394s) [seg_pos={0 0}, upload=2817]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:25.127534016 (3h2m40.908467396s) [upload=2817]: Done uploading segments... Jul 27 00:54:25.127566856 (3h2m40.908500238s) [upload=2817]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 424.254536ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXXX] duration: 1m0.028398616s FAIL (2025-07-27 00:54:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3h1m32.393441024s) [upload=2808]: Got next segment. Inline: false Jul 27 00:53:16.612589750 (3h1m32.393523138s) [upload=2808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:53:16.612597496 (3h1m32.393530889s) [upload=2808]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:53:16.649453991 (3h1m32.430387387s) [seg_pos={0 0}, upload=2808]: Begin upload segment... Jul 27 00:53:16.649537526 (3h1m32.430470913s) [seg_pos={0 0}, upload=2808]: Done begin upload segment. Jul 27 00:53:16.649568295 (3h1m32.43050168s) [noise=true , address=104.36.82.39:4743 , piece_id=XJ7INQTUC3HXEG7EIXP56SMUA6UBN25K4L27MW6TTEET7A26U2BQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.649584395 (3h1m32.430517781s) [noise=true , address=79.127.205.245:4247 , piece_id=NM42JKIQ5SRNHPULLTBNYZUUWOXGPHNVNUA3FHAYTIYXA5ML7CTQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.650003134 (3h1m32.430936523s) [noise=true , address=79.127.205.235:4685 , piece_id=NARZAGTAYZBJV34FGOXLXO6XWPYYH6LATWBRZDCY22CSK32MKJYQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.650214987 (3h1m32.431148374s) [noise=true , address=79.127.220.101:4041 , piece_id=27NTVODN6LA7B5I2ND3DJEMOBAFNLWMBDNE3KM7SELEPS5RAKQ6Q, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.651266754 (3h1m32.432200135s) [noise=true , address=77.90.31.18:4836 , piece_id=DQ647FBVVXCVQBO4XB2DD4IX7ULLQBPQI3FJ3A72SMT5JNRB37KA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.651653916 (3h1m32.432587297s) [noise=true , address=77.90.31.10:4540 , piece_id=PJBXE3SOHIJCHM2RDBUPEVZKHPOF2ZWUWE5HLBPVBGX4I75PF2CA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.652174751 (3h1m32.433108141s) [noise=true , address=104.36.82.47:4545 , piece_id=SPRBQSURNHKI2BD7JLVFUHR27BXFLHL6U6PR4Z7W3VFIVPDELZUQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.652578950 (3h1m32.433512362s) [noise=true , address=79.127.201.215:4346 , piece_id=4HJVFIYTRAQE2ZPSBAQCNMWFSRDPDHO6DAXIJQ6VDO7ZDPWVOYLQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.652979664 (3h1m32.433913164s) [noise=true , address=79.127.205.234:4620 , piece_id=HZMI5CF7G2LR7FTO2GUK2NIEQFRSG4PYHKM3JEVXTRAHUDT6WNQA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.653885996 (3h1m32.434819382s) [noise=true , address=156.146.43.225:4448 , piece_id=OD5ZLEZV45NXSI5SEY7RXPZCJMZRZM3PGJOQAPPQTALZEM54JRNQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.654332656 (3h1m32.435266041s) [noise=true , address=104.36.82.53:4147 , piece_id=UVQJULEP55JC4J7DZSLTL3BW4EXHWN4UTMDVVFDWLKUXHWCVBJGA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.654731150 (3h1m32.435664537s) [noise=true , address=104.36.82.4:4824 , piece_id=4BKCZFOJ7QRQCHUC7TH6GOSVTS42L7D25W2VDX7QJPZ6SHXLSP2A, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.655250729 (3h1m32.436184114s) [noise=true , address=45.91.116.11:4130 , piece_id=XCGYF3T5O35RESUSB6ANU43GO56BIHT7P7PDUMQVN3MDRF56HG6Q, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.655756052 (3h1m32.436690073s) [noise=true , address=79.127.205.233:4650 , piece_id=SLEMW5GOW3UVFEKJGLFVZLBCX5GY2TAREIMMNFKS5GCQ3ZG4TWBQ, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.656324680 (3h1m32.437258065s) [noise=true , address=104.36.82.55:4199 , piece_id=7S3U7VJOA7ABYL7I5Y3FPCKLSFJW26U3YF4ZBSZUBXWHLU5IADSA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.656860605 (3h1m32.437794036s) [noise=true , address=104.36.82.46:4489 , piece_id=HMQHDGRC5NJ5FUTFA4BZWWBVFKXSAP3A3CN3XB3HYPUAKOPJAI7A, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.657397946 (3h1m32.438331425s) [noise=true , address=45.91.116.10:4103 , piece_id=ZC7G3BF2HFKXGTMMRZRL2IISDW6HVEEJV7LQ3334VV7ENJB6IJGA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.657883857 (3h1m32.438817241s) [noise=true , address=79.127.205.246:4258 , piece_id=UDV73BKE7HKWHYVDY73PQKYIJU64FVBS4TMECKC7SZARTV4TQUTA, seg_pos={0 0}, upload=2808]: Uploading piece... Jul 27 00:53:16.728714078 (3h1m32.509647453s) [upload=2808]: Next returned nil segment Jul 27 00:53:16.728732317 (3h1m32.509665697s) [upload=2808]: Waiting for error group managing segments... Jul 27 00:53:16.756775898 (3h1m32.537709266s) [noise=true , address=77.90.31.18:4836 , piece_id=DQ647FBVVXCVQBO4XB2DD4IX7ULLQBPQI3FJ3A72SMT5JNRB37KA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:16.860381019 (3h1m32.641314391s) [noise=true , address=77.90.31.10:4540 , piece_id=PJBXE3SOHIJCHM2RDBUPEVZKHPOF2ZWUWE5HLBPVBGX4I75PF2CA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:17.225416178 (3h1m33.00634962s) [noise=true , address=79.127.205.235:4685 , piece_id=NARZAGTAYZBJV34FGOXLXO6XWPYYH6LATWBRZDCY22CSK32MKJYQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:17.249550005 (3h1m33.030483383s) [noise=true , address=79.127.205.234:4620 , piece_id=HZMI5CF7G2LR7FTO2GUK2NIEQFRSG4PYHKM3JEVXTRAHUDT6WNQA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:17.258876279 (3h1m33.039809661s) [noise=true , address=79.127.205.245:4247 , piece_id=NM42JKIQ5SRNHPULLTBNYZUUWOXGPHNVNUA3FHAYTIYXA5ML7CTQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:17.260264547 (3h1m33.041197925s) [noise=true , address=79.127.205.246:4258 , piece_id=UDV73BKE7HKWHYVDY73PQKYIJU64FVBS4TMECKC7SZARTV4TQUTA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:18.886175063 (3h1m34.667108432s) [noise=true , address=79.127.220.101:4041 , piece_id=27NTVODN6LA7B5I2ND3DJEMOBAFNLWMBDNE3KM7SELEPS5RAKQ6Q, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:19.703702032 (3h1m35.484635417s) [noise=true , address=79.127.205.233:4650 , piece_id=SLEMW5GOW3UVFEKJGLFVZLBCX5GY2TAREIMMNFKS5GCQ3ZG4TWBQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:21.159547004 (3h1m36.94048038s) [noise=true , address=45.91.116.10:4103 , piece_id=ZC7G3BF2HFKXGTMMRZRL2IISDW6HVEEJV7LQ3334VV7ENJB6IJGA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:22.605696066 (3h1m38.386629451s) [noise=true , address=156.146.43.225:4448 , piece_id=OD5ZLEZV45NXSI5SEY7RXPZCJMZRZM3PGJOQAPPQTALZEM54JRNQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:26.226866633 (3h1m42.007800011s) [noise=true , address=45.91.116.11:4130 , piece_id=XCGYF3T5O35RESUSB6ANU43GO56BIHT7P7PDUMQVN3MDRF56HG6Q, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:53:40.145062262 (3h1m55.925995639s) [noise=true , address=79.127.201.215:4346 , piece_id=4HJVFIYTRAQE2ZPSBAQCNMWFSRDPDHO6DAXIJQ6VDO7ZDPWVOYLQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: <nil> Jul 27 00:54:16.585011327 (3h2m32.365944713s) [noise=true , address=104.36.82.47:4545 , piece_id=SPRBQSURNHKI2BD7JLVFUHR27BXFLHL6U6PR4Z7W3VFIVPDELZUQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585122715 (3h2m32.3660561s) [noise=true , address=104.36.82.55:4199 , piece_id=7S3U7VJOA7ABYL7I5Y3FPCKLSFJW26U3YF4ZBSZUBXWHLU5IADSA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585207431 (3h2m32.366140814s) [noise=true , address=104.36.82.46:4489 , piece_id=HMQHDGRC5NJ5FUTFA4BZWWBVFKXSAP3A3CN3XB3HYPUAKOPJAI7A, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: write tcp 159.223.143.39:34368->104.36.82.46:4489: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585457067 (3h2m32.366390447s) [noise=true , address=104.36.82.53:4147 , piece_id=UVQJULEP55JC4J7DZSLTL3BW4EXHWN4UTMDVVFDWLKUXHWCVBJGA, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585547307 (3h2m32.366480693s) [noise=true , address=104.36.82.4:4824 , piece_id=4BKCZFOJ7QRQCHUC7TH6GOSVTS42L7D25W2VDX7QJPZ6SHXLSP2A, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585628510 (3h2m32.366561891s) [noise=true , address=104.36.82.39:4743 , piece_id=XJ7INQTUC3HXEG7EIXP56SMUA6UBN25K4L27MW6TTEET7A26U2BQ, seg_pos={0 0}, upload=2808]: Done uploading piece. err: ecclient: upload failed (node:12XHeYbbHG3TpFDS5ZogLPHZRaYuY7SA5seRU72UoR21JCVcYU9, address:104.36.82.39:4743): protocol: write tcp 159.223.143.39:56318->104.36.82.39:4743: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:16.585703365 (3h2m32.366636753s) [seg_pos={0 0}, upload=2808]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:54:16.607557708 (3h2m32.388491078s) [upload=2808]: Done uploading segments... Jul 27 00:54:16.607582560 (3h2m32.388515927s) [upload=2808]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.028562499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......................X......X.....] duration: 294.986478ms FAIL (2025-07-27 00:54:04 UTC) orbiter warning: uplink: too many requests Duration: 295.018343ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [................XX....X......XX.XXXX...X....................] duration: 1.897960778s FAIL (2025-07-27 00:53:59 UTC) orbiter warning: context deadline exceeded Duration: 1.897989451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [X....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X....] duration: 758.890929ms FAIL (2025-07-27 00:53:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 758.92411ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................XX....X......XX.XXXX...X...................] duration: 8.58084523s FAIL (2025-07-27 00:53:31 UTC) orbiter warning: context deadline exceeded Duration: 8.580866745s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XXX] duration: 1m0.035707398s FAIL (2025-07-27 00:53:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Jul 27 00:52:16.541605866 (3h0m32.322539247s) [noise=true , address=104.36.82.44:4890 , piece_id=77C6Q4OEGEWB22JNNUKF4CBIPIRB7GWSV55MHCDYJRWCFCKVACAQ, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.542009985 (3h0m32.32294336s) [noise=true , address=45.91.116.14:4206 , piece_id=DELWABDHQXKCKIUS7T7UW4K7HOKIUTBIMVKEUIQLK4A2SHJWLDBQ, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.542036884 (3h0m32.322970262s) [noise=true , address=104.36.82.54:4159 , piece_id=BBL723YAPHXL7PQKTPH5YPWRRTBR4EJZPOHCVTJXSRRWLYMG3UWA, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.542345687 (3h0m32.323279114s) [noise=true , address=79.127.205.246:4272 , piece_id=5K75AZVA4FJV4EKSHWLCBOEKCSKC3TO6HP2ILX7CZHBNKXYTBQ5Q, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.542353277 (3h0m32.323286662s) [noise=true , address=104.36.82.46:4510 , piece_id=ABEBMZRANWFRGZKZJXLF5JGLFG2FUEJAEUTXUKZSILH2YWWBADBQ, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.542706924 (3h0m32.323640303s) [noise=true , address=64.49.13.140:4514 , piece_id=6VG6M5OOQO6EAF2T737MQJHRHB2S7Z3G3AHME3SCAXW3I3XO3ZNQ, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.542727378 (3h0m32.323660757s) [noise=true , address=104.36.82.47:4541 , piece_id=LUHHFQGMOBWEUI54QV7K32NTCF2G4OAZ54LX35XIYPDFWWVG5IYQ, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.555713806 (3h0m32.336647173s) [noise=true , address=156.146.43.225:4448 , piece_id=X5FI5EGY46KRDUBJ2R3ZXQLSPJ6G5ISGBBHPRR5EXNJPTKDYFMTA, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.555881146 (3h0m32.336814528s) [noise=true , address=79.127.220.101:4041 , piece_id=BX3TLDATO2VJFVXA7XM3DY3VS244DZAT3GPETIV7TLMEGBKTRZ2A, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.556539746 (3h0m32.337473135s) [noise=true , address=79.127.205.234:4603 , piece_id=BBXATWRLFI5Z4AIBP4RCQ5RTSPSYBKY3FS2XHAGQLRU5RKQ7JCGA, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.557747906 (3h0m32.338681293s) [noise=true , address=77.90.31.21:4908 , piece_id=FMSKN3AKOWU2ECSKIKKY7OSJ4E2SDWJHBR272BBW37T63TJ4ZFAA, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.561528964 (3h0m32.342462354s) [noise=true , address=104.36.82.12:4166 , piece_id=H54CPDQ3KNOWTKURECPJT3X2VYI36E6NL3EUEZJ7H3EVODGEBPVA, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.565053788 (3h0m32.345987174s) [noise=true , address=79.127.201.215:4344 , piece_id=2B24W7RIFFW3ZO3AGP3AGEIUBTPRBALJLHVMHMPDA2L7UHCJIM7Q, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.568214599 (3h0m32.349147984s) [noise=true , address=104.36.82.55:4215 , piece_id=WVAAAKIAZWLFTF34LXQA4SDQGSOZNK2LP47ZCXTZF6G34EVWKRPQ, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.571152306 (3h0m32.352085695s) [noise=true , address=104.36.82.53:4135 , piece_id=4FVLZXFBOFULYTZTUWH4AFHQ5VSGVRVQR2EZS3K3ZFODLXHENHIA, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.573608974 (3h0m32.354542362s) [noise=true , address=79.127.205.235:4683 , piece_id=V2HXJDU4UAYYDSQIE7OOLCDMJPPMQUKQSUIJASJ6PAQRDMTSL7YA, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.576340891 (3h0m32.35727428s) [noise=true , address=77.90.31.19:4871 , piece_id=2RBWQFQSWOAPXXPPIOWWWH4P5E7JV72WFDSVRKR26HBJEIANTP4Q, seg_pos={0 0}, upload=2789]: Uploading piece... Jul 27 00:52:16.603456432 (3h0m32.384389806s) [noise=true , address=77.90.31.19:4871 , piece_id=2RBWQFQSWOAPXXPPIOWWWH4P5E7JV72WFDSVRKR26HBJEIANTP4Q, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: storage node overloaded, request limit: 2000 Jul 27 00:52:16.631059759 (3h0m32.411993145s) [upload=2789]: Next returned nil segment Jul 27 00:52:16.631085445 (3h0m32.412018824s) [upload=2789]: Waiting for error group managing segments... Jul 27 00:52:16.688228112 (3h0m32.469161487s) [noise=true , address=77.90.31.21:4908 , piece_id=FMSKN3AKOWU2ECSKIKKY7OSJ4E2SDWJHBR272BBW37T63TJ4ZFAA, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:52:16.852973561 (3h0m32.633906931s) [noise=true , address=79.127.220.101:4041 , piece_id=BX3TLDATO2VJFVXA7XM3DY3VS244DZAT3GPETIV7TLMEGBKTRZ2A, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:52:17.135298431 (3h0m32.916231799s) [noise=true , address=79.127.205.234:4603 , piece_id=BBXATWRLFI5Z4AIBP4RCQ5RTSPSYBKY3FS2XHAGQLRU5RKQ7JCGA, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:52:17.144901309 (3h0m32.925834686s) [noise=true , address=79.127.205.245:4242 , piece_id=LFLMBKQRUOHE6FEZ7AECGYBQR5JB5HB77UNRI6DZXKVA5SGXX4IQ, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:52:17.154162036 (3h0m32.935095413s) [noise=true , address=79.127.205.246:4272 , piece_id=5K75AZVA4FJV4EKSHWLCBOEKCSKC3TO6HP2ILX7CZHBNKXYTBQ5Q, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:52:17.162768333 (3h0m32.943701702s) [noise=true , address=79.127.205.235:4683 , piece_id=V2HXJDU4UAYYDSQIE7OOLCDMJPPMQUKQSUIJASJ6PAQRDMTSL7YA, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:52:19.633099122 (3h0m35.41403249s) [noise=true , address=156.146.43.225:4448 , piece_id=X5FI5EGY46KRDUBJ2R3ZXQLSPJ6G5ISGBBHPRR5EXNJPTKDYFMTA, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:52:20.320295426 (3h0m36.10122879s) [noise=true , address=64.49.13.140:4514 , piece_id=6VG6M5OOQO6EAF2T737MQJHRHB2S7Z3G3AHME3SCAXW3I3XO3ZNQ, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 27 00:52:20.376230928 (3h0m36.15716431s) [noise=true , address=45.91.116.14:4206 , piece_id=DELWABDHQXKCKIUS7T7UW4K7HOKIUTBIMVKEUIQLK4A2SHJWLDBQ, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:52:41.919101425 (3h0m57.700034786s) [noise=true , address=79.127.201.215:4344 , piece_id=2B24W7RIFFW3ZO3AGP3AGEIUBTPRBALJLHVMHMPDA2L7UHCJIM7Q, seg_pos={0 0}, upload=2789]: Done uploading piece. err: <nil> Jul 27 00:53:16.476470449 (3h1m32.257404032s) [noise=true , address=104.36.82.46:4510 , piece_id=ABEBMZRANWFRGZKZJXLF5JGLFG2FUEJAEUTXUKZSILH2YWWBADBQ, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: write tcp 159.223.143.39:59562->104.36.82.46:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:16.476562866 (3h1m32.257496242s) [noise=true , address=104.36.82.47:4541 , piece_id=LUHHFQGMOBWEUI54QV7K32NTCF2G4OAZ54LX35XIYPDFWWVG5IYQ, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: write tcp 159.223.143.39:54610->104.36.82.47:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:16.476643651 (3h1m32.257577014s) [noise=true , address=104.36.82.12:4166 , piece_id=H54CPDQ3KNOWTKURECPJT3X2VYI36E6NL3EUEZJ7H3EVODGEBPVA, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: write tcp 159.223.143.39:43368->104.36.82.12:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:16.476718904 (3h1m32.257652283s) [noise=true , address=104.36.82.55:4215 , piece_id=WVAAAKIAZWLFTF34LXQA4SDQGSOZNK2LP47ZCXTZF6G34EVWKRPQ, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:35574->104.36.82.55:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:16.476790971 (3h1m32.257724347s) [noise=true , address=104.36.82.54:4159 , piece_id=BBL723YAPHXL7PQKTPH5YPWRRTBR4EJZPOHCVTJXSRRWLYMG3UWA, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:104.36.82.54:4159): protocol: rpc: dial tcp 104.36.82.54:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:16.476860953 (3h1m32.257794333s) [noise=true , address=104.36.82.53:4135 , piece_id=4FVLZXFBOFULYTZTUWH4AFHQ5VSGVRVQR2EZS3K3ZFODLXHENHIA, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 159.223.143.39:36510->104.36.82.53:4135: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:16.476935446 (3h1m32.257868825s) [noise=true , address=104.36.82.44:4890 , piece_id=77C6Q4OEGEWB22JNNUKF4CBIPIRB7GWSV55MHCDYJRWCFCKVACAQ, seg_pos={0 0}, upload=2789]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: write tcp 159.223.143.39:44260->104.36.82.44:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:16.476963979 (3h1m32.257897364s) [seg_pos={0 0}, upload=2789]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:53:16.508578704 (3h1m32.289512085s) [upload=2789]: Done uploading segments... Jul 27 00:53:16.508610690 (3h1m32.289544076s) [upload=2789]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.035730847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X...] duration: 556.906982ms FAIL (2025-07-27 00:53:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 556.938035ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......................X......X....] duration: 255.087085ms FAIL (2025-07-27 00:53:00 UTC) orbiter warning: uplink: too many requests Duration: 255.203741ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................X...............X...X...............X...] duration: 2.580796249s FAIL (2025-07-27 00:52:59 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.580818779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX.] duration: 2.065903526s FAIL (2025-07-27 00:52:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 2961628s) [upload=2788]: Starting upload Jul 27 00:52:11.022161434 (3h0m26.8030948s) [upload=2788]: Done starting upload Jul 27 00:52:11.022261250 (3h0m26.803194638s) [upload=2788]: Uploading segments... Jul 27 00:52:11.022614959 (3h0m26.803548334s) [upload=2788]: Got next segment. Inline: false Jul 27 00:52:11.022733506 (3h0m26.803666887s) [upload=2788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:11.022743066 (3h0m26.803676453s) [upload=2788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:11.056009048 (3h0m26.836942441s) [seg_pos={0 0}, upload=2788]: Begin upload segment... Jul 27 00:52:11.056060801 (3h0m26.836994181s) [seg_pos={0 0}, upload=2788]: Done begin upload segment. Jul 27 00:52:11.056104451 (3h0m26.837037835s) [noise=true , address=104.36.82.8:4969 , piece_id=SDC6NQJIC252MKPVALCJYCMX4PRAFQHVUC4QTLLHLBDTR4KG66WQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056138747 (3h0m26.83707213s) [noise=true , address=104.36.82.10:4077 , piece_id=2BLUVQCGW2DWD776VWAI23BIEPWF7W2V5VL6JXXLBNNFDFKL3T7Q, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056249327 (3h0m26.837182705s) [noise=true , address=45.91.116.5:4299 , piece_id=62ZVPPP37OTFUXXJ645UHXQA4GPLKA5ILHP4XEQZDSEFTQCKH3EQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056601113 (3h0m26.837534492s) [noise=true , address=77.90.31.17:4771 , piece_id=DGXQJAZCX3OOPDJCJHUZRGVWTKWWHVEYG7TNKUNBRHFMQH7KQZGA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056662756 (3h0m26.837596141s) [noise=true , address=45.91.116.12:4227 , piece_id=BX5ZBDFEJYOA6LMOW5C5EADZMI5HFN5M2GJLQBMTRXWVV7AQOVAQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056942778 (3h0m26.837876171s) [noise=true , address=104.36.82.11:4113 , piece_id=K274GULG22MLFFSFJVEXPTSMMCHMXNWNYLXPDTFNWLBG6CKBRKIQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.057138056 (3h0m26.838071445s) [noise=true , address=104.36.82.16:4295 , piece_id=ZQP6V5URWAQPGDOOTYUHSGE2PLNPOU5BFEXZPT326NEPKM7LJHJA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.057323643 (3h0m26.838257028s) [noise=true , address=104.36.82.15:4257 , piece_id=LHY37LK6VOUWG4C65ZJUA43WYHW6TG7CRAL7PMNMETKZA4UQSMVQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.057688425 (3h0m26.838621803s) [noise=true , address=77.90.31.19:4841 , piece_id=72OY2B3QRGLPP77QQBLFMMFFGDSUXZPWAJGTVOH5NWVK6Y4KQCTA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.057736659 (3h0m26.838670037s) [noise=true , address=104.36.82.14:4222 , piece_id=MMU5TVHF5PGAGK3E3UO3OQDEU4WL4JA2OTYQOZMFSGX256XVQZIQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.058076876 (3h0m26.839010261s) [noise=true , address=77.90.31.14:4661 , piece_id=H3UODBNOXJISB5DRPTDUM6BYQ6MCLG2HZTKW2YQ2XKJD52IUYUGA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.058247316 (3h0m26.839180703s) [noise=true , address=77.90.31.10:4517 , piece_id=4IWVGJMDKEINHHFMCZ7JTRZIWTXN27JGWANF2TEBSWD7ZBA7X3WA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.058451286 (3h0m26.839384669s) [noise=true , address=104.36.82.3:4767 , piece_id=5WO5AE5CXVEHS2RCSV4L5TBEQFB2YR2FTZN243PCMOGWPGEOM3XQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.058745804 (3h0m26.83967919s) [noise=true , address=45.91.116.6:4975 , piece_id=MJIDJ64JZ3IWUGHRFJ6AXY2SZ5LILOMIR6DDMOTNVYYMUIHG3HDA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.059326406 (3h0m26.840259801s) [noise=true , address=77.90.31.17:4772 , piece_id=UIJ2ETEGDMPI6G5IVRWK2KNYQ3DR62GWWZH2FQVFERCOESDMBVXQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.059832867 (3h0m26.840766256s) [noise=true , address=45.91.116.11:4117 , piece_id=UX7QOXERZPAKBNYQT2OVSU2IQF7AFSCR6JPTATJI3WMRWA6ZYTXA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.138915743 (3h0m26.919849128s) [upload=2788]: Next returned nil segment Jul 27 00:52:11.138935636 (3h0m26.919869024s) [upload=2788]: Waiting for error group managing segments... Jul 27 00:52:11.144441617 (3h0m26.925374995s) [noise=true , address=45.91.116.11:4117 , piece_id=UX7QOXERZPAKBNYQT2OVSU2IQF7AFSCR6JPTATJI3WMRWA6ZYTXA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.144930568 (3h0m26.925863938s) [noise=true , address=77.90.31.10:4517 , piece_id=4IWVGJMDKEINHHFMCZ7JTRZIWTXN27JGWANF2TEBSWD7ZBA7X3WA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.145486606 (3h0m26.926419974s) [noise=true , address=45.91.116.6:4975 , piece_id=MJIDJ64JZ3IWUGHRFJ6AXY2SZ5LILOMIR6DDMOTNVYYMUIHG3HDA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.145917931 (3h0m26.926851296s) [noise=true , address=45.91.116.12:4227 , piece_id=BX5ZBDFEJYOA6LMOW5C5EADZMI5HFN5M2GJLQBMTRXWVV7AQOVAQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.146360944 (3h0m26.927294309s) [noise=true , address=45.91.116.5:4299 , piece_id=62ZVPPP37OTFUXXJ645UHXQA4GPLKA5ILHP4XEQZDSEFTQCKH3EQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.146953417 (3h0m26.927886793s) [noise=true , address=77.90.31.14:4661 , piece_id=H3UODBNOXJISB5DRPTDUM6BYQ6MCLG2HZTKW2YQ2XKJD52IUYUGA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:12.529571583 (3h0m28.31050495s) [noise=true , address=77.90.31.19:4841 , piece_id=72OY2B3QRGLPP77QQBLFMMFFGDSUXZPWAJGTVOH5NWVK6Y4KQCTA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:40.988871298 (3h0m56.76980468s) [noise=true , address=104.36.82.10:4077 , piece_id=2BLUVQCGW2DWD776VWAI23BIEPWF7W2V5VL6JXXLBNNFDFKL3T7Q, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989009570 (3h0m56.769942946s) [noise=true , address=104.36.82.15:4257 , piece_id=LHY37LK6VOUWG4C65ZJUA43WYHW6TG7CRAL7PMNMETKZA4UQSMVQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:52300->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989022166 (3h0m56.769955542s) [noise=true , address=104.36.82.14:4222 , piece_id=MMU5TVHF5PGAGK3E3UO3OQDEU4WL4JA2OTYQOZMFSGX256XVQZIQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989091150 (3h0m56.770024525s) [noise=true , address=104.36.82.3:4767 , piece_id=5WO5AE5CXVEHS2RCSV4L5TBEQFB2YR2FTZN243PCMOGWPGEOM3XQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989118426 (3h0m56.770051802s) [noise=true , address=104.36.82.8:4969 , piece_id=SDC6NQJIC252MKPVALCJYCMX4PRAFQHVUC4QTLLHLBDTR4KG66WQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:52:40.989176180 (3h0m56.770109556s) [noise=true , address=77.90.31.17:4772 , piece_id=UIJ2ETEGDMPI6G5IVRWK2KNYQ3DR62GWWZH2FQVFERCOESDMBVXQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989247965 (3h0m56.770181331s) [noise=true , address=104.36.82.11:4113 , piece_id=K274GULG22MLFFSFJVEXPTSMMCHMXNWNYLXPDTFNWLBG6CKBRKIQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:38584->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989283937 (3h0m56.770217302s) [noise=true , address=104.36.82.16:4295 , piece_id=ZQP6V5URWAQPGDOOTYUHSGE2PLNPOU5BFEXZPT326NEPKM7LJHJA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989292818 (3h0m56.770226187s) [noise=true , address=77.90.31.17:4771 , piece_id=DGXQJAZCX3OOPDJCJHUZRGVWTKWWHVEYG7TNKUNBRHFMQH7KQZGA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989323311 (3h0m56.770256688s) [seg_pos={0 0}, upload=2788]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 27 00:52:41.014408621 (3h0m56.795342006s) [upload=2788]: Done uploading segments... Jul 27 00:52:41.014471720 (3h0m56.795405105s) [upload=2788]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 2.065930886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXXX] duration: 30.027979021s FAIL (2025-07-27 00:52:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 2961628s) [upload=2788]: Starting upload Jul 27 00:52:11.022161434 (3h0m26.8030948s) [upload=2788]: Done starting upload Jul 27 00:52:11.022261250 (3h0m26.803194638s) [upload=2788]: Uploading segments... Jul 27 00:52:11.022614959 (3h0m26.803548334s) [upload=2788]: Got next segment. Inline: false Jul 27 00:52:11.022733506 (3h0m26.803666887s) [upload=2788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:11.022743066 (3h0m26.803676453s) [upload=2788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:11.056009048 (3h0m26.836942441s) [seg_pos={0 0}, upload=2788]: Begin upload segment... Jul 27 00:52:11.056060801 (3h0m26.836994181s) [seg_pos={0 0}, upload=2788]: Done begin upload segment. Jul 27 00:52:11.056104451 (3h0m26.837037835s) [noise=true , address=104.36.82.8:4969 , piece_id=SDC6NQJIC252MKPVALCJYCMX4PRAFQHVUC4QTLLHLBDTR4KG66WQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056138747 (3h0m26.83707213s) [noise=true , address=104.36.82.10:4077 , piece_id=2BLUVQCGW2DWD776VWAI23BIEPWF7W2V5VL6JXXLBNNFDFKL3T7Q, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056249327 (3h0m26.837182705s) [noise=true , address=45.91.116.5:4299 , piece_id=62ZVPPP37OTFUXXJ645UHXQA4GPLKA5ILHP4XEQZDSEFTQCKH3EQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056601113 (3h0m26.837534492s) [noise=true , address=77.90.31.17:4771 , piece_id=DGXQJAZCX3OOPDJCJHUZRGVWTKWWHVEYG7TNKUNBRHFMQH7KQZGA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056662756 (3h0m26.837596141s) [noise=true , address=45.91.116.12:4227 , piece_id=BX5ZBDFEJYOA6LMOW5C5EADZMI5HFN5M2GJLQBMTRXWVV7AQOVAQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.056942778 (3h0m26.837876171s) [noise=true , address=104.36.82.11:4113 , piece_id=K274GULG22MLFFSFJVEXPTSMMCHMXNWNYLXPDTFNWLBG6CKBRKIQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.057138056 (3h0m26.838071445s) [noise=true , address=104.36.82.16:4295 , piece_id=ZQP6V5URWAQPGDOOTYUHSGE2PLNPOU5BFEXZPT326NEPKM7LJHJA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.057323643 (3h0m26.838257028s) [noise=true , address=104.36.82.15:4257 , piece_id=LHY37LK6VOUWG4C65ZJUA43WYHW6TG7CRAL7PMNMETKZA4UQSMVQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.057688425 (3h0m26.838621803s) [noise=true , address=77.90.31.19:4841 , piece_id=72OY2B3QRGLPP77QQBLFMMFFGDSUXZPWAJGTVOH5NWVK6Y4KQCTA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.057736659 (3h0m26.838670037s) [noise=true , address=104.36.82.14:4222 , piece_id=MMU5TVHF5PGAGK3E3UO3OQDEU4WL4JA2OTYQOZMFSGX256XVQZIQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.058076876 (3h0m26.839010261s) [noise=true , address=77.90.31.14:4661 , piece_id=H3UODBNOXJISB5DRPTDUM6BYQ6MCLG2HZTKW2YQ2XKJD52IUYUGA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.058247316 (3h0m26.839180703s) [noise=true , address=77.90.31.10:4517 , piece_id=4IWVGJMDKEINHHFMCZ7JTRZIWTXN27JGWANF2TEBSWD7ZBA7X3WA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.058451286 (3h0m26.839384669s) [noise=true , address=104.36.82.3:4767 , piece_id=5WO5AE5CXVEHS2RCSV4L5TBEQFB2YR2FTZN243PCMOGWPGEOM3XQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.058745804 (3h0m26.83967919s) [noise=true , address=45.91.116.6:4975 , piece_id=MJIDJ64JZ3IWUGHRFJ6AXY2SZ5LILOMIR6DDMOTNVYYMUIHG3HDA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.059326406 (3h0m26.840259801s) [noise=true , address=77.90.31.17:4772 , piece_id=UIJ2ETEGDMPI6G5IVRWK2KNYQ3DR62GWWZH2FQVFERCOESDMBVXQ, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.059832867 (3h0m26.840766256s) [noise=true , address=45.91.116.11:4117 , piece_id=UX7QOXERZPAKBNYQT2OVSU2IQF7AFSCR6JPTATJI3WMRWA6ZYTXA, seg_pos={0 0}, upload=2788]: Uploading piece... Jul 27 00:52:11.138915743 (3h0m26.919849128s) [upload=2788]: Next returned nil segment Jul 27 00:52:11.138935636 (3h0m26.919869024s) [upload=2788]: Waiting for error group managing segments... Jul 27 00:52:11.144441617 (3h0m26.925374995s) [noise=true , address=45.91.116.11:4117 , piece_id=UX7QOXERZPAKBNYQT2OVSU2IQF7AFSCR6JPTATJI3WMRWA6ZYTXA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.144930568 (3h0m26.925863938s) [noise=true , address=77.90.31.10:4517 , piece_id=4IWVGJMDKEINHHFMCZ7JTRZIWTXN27JGWANF2TEBSWD7ZBA7X3WA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.145486606 (3h0m26.926419974s) [noise=true , address=45.91.116.6:4975 , piece_id=MJIDJ64JZ3IWUGHRFJ6AXY2SZ5LILOMIR6DDMOTNVYYMUIHG3HDA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.145917931 (3h0m26.926851296s) [noise=true , address=45.91.116.12:4227 , piece_id=BX5ZBDFEJYOA6LMOW5C5EADZMI5HFN5M2GJLQBMTRXWVV7AQOVAQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.146360944 (3h0m26.927294309s) [noise=true , address=45.91.116.5:4299 , piece_id=62ZVPPP37OTFUXXJ645UHXQA4GPLKA5ILHP4XEQZDSEFTQCKH3EQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:11.146953417 (3h0m26.927886793s) [noise=true , address=77.90.31.14:4661 , piece_id=H3UODBNOXJISB5DRPTDUM6BYQ6MCLG2HZTKW2YQ2XKJD52IUYUGA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:12.529571583 (3h0m28.31050495s) [noise=true , address=77.90.31.19:4841 , piece_id=72OY2B3QRGLPP77QQBLFMMFFGDSUXZPWAJGTVOH5NWVK6Y4KQCTA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: <nil> Jul 27 00:52:40.988871298 (3h0m56.76980468s) [noise=true , address=104.36.82.10:4077 , piece_id=2BLUVQCGW2DWD776VWAI23BIEPWF7W2V5VL6JXXLBNNFDFKL3T7Q, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989009570 (3h0m56.769942946s) [noise=true , address=104.36.82.15:4257 , piece_id=LHY37LK6VOUWG4C65ZJUA43WYHW6TG7CRAL7PMNMETKZA4UQSMVQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:52300->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989022166 (3h0m56.769955542s) [noise=true , address=104.36.82.14:4222 , piece_id=MMU5TVHF5PGAGK3E3UO3OQDEU4WL4JA2OTYQOZMFSGX256XVQZIQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989091150 (3h0m56.770024525s) [noise=true , address=104.36.82.3:4767 , piece_id=5WO5AE5CXVEHS2RCSV4L5TBEQFB2YR2FTZN243PCMOGWPGEOM3XQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989118426 (3h0m56.770051802s) [noise=true , address=104.36.82.8:4969 , piece_id=SDC6NQJIC252MKPVALCJYCMX4PRAFQHVUC4QTLLHLBDTR4KG66WQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:52:40.989176180 (3h0m56.770109556s) [noise=true , address=77.90.31.17:4772 , piece_id=UIJ2ETEGDMPI6G5IVRWK2KNYQ3DR62GWWZH2FQVFERCOESDMBVXQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989247965 (3h0m56.770181331s) [noise=true , address=104.36.82.11:4113 , piece_id=K274GULG22MLFFSFJVEXPTSMMCHMXNWNYLXPDTFNWLBG6CKBRKIQ, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:38584->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989283937 (3h0m56.770217302s) [noise=true , address=104.36.82.16:4295 , piece_id=ZQP6V5URWAQPGDOOTYUHSGE2PLNPOU5BFEXZPT326NEPKM7LJHJA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989292818 (3h0m56.770226187s) [noise=true , address=77.90.31.17:4771 , piece_id=DGXQJAZCX3OOPDJCJHUZRGVWTKWWHVEYG7TNKUNBRHFMQH7KQZGA, seg_pos={0 0}, upload=2788]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:40.989323311 (3h0m56.770256688s) [seg_pos={0 0}, upload=2788]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 27 00:52:41.014408621 (3h0m56.795342006s) [upload=2788]: Done uploading segments... Jul 27 00:52:41.014471720 (3h0m56.795405105s) [upload=2788]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.028029012s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................X...............X...X...............X..] duration: 23.39419498s FAIL (2025-07-27 00:52:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.394561412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.................XX....X......XX.XXXX...X..................] duration: 496.088851ms FAIL (2025-07-27 00:52:27 UTC) orbiter warning: context deadline exceeded Duration: 496.117598ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........XX] duration: 1m0.029867601s FAIL (2025-07-27 00:52:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== lush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:51:16.464975755 (2h59m32.245909139s) [seg_pos={0 0}, upload=2772]: Begin upload segment... Jul 27 00:51:16.465056495 (2h59m32.245989934s) [seg_pos={0 0}, upload=2772]: Done begin upload segment. Jul 27 00:51:16.465090402 (2h59m32.246023789s) [noise=true , address=79.127.201.215:4356 , piece_id=7FN5YUCYCWM4YZWFZNB3DCW66445K5G4N4XTXACTUD5FKIM6O7VQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.465115685 (2h59m32.24604907s) [noise=true , address=104.36.82.10:4109 , piece_id=Y43CNHUPXVHVKZYOVOINYZZJYCH4MATGSPJX24VTCESSVTK62BZQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.465559435 (2h59m32.246492818s) [noise=true , address=64.49.13.138:4483 , piece_id=SSQZHGTPJK527YBIDR26IR3FQPXLCR5MLIM2KEHT64QBAPWVF2QQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.465721796 (2h59m32.246655182s) [noise=true , address=79.127.205.233:4656 , piece_id=KCYAWSPDXNB7UIVGLARV2FTM3LBCV2VDRRPNMKJTITT2K2BK7EUQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.465911104 (2h59m32.246844479s) [noise=true , address=79.127.205.234:4601 , piece_id=3WPXFNT3NBPBZMNSBGVM53C7EDZSZ4EXN4MNQNLPDVVOUCTEAFJQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.466211453 (2h59m32.247144819s) [noise=true , address=64.49.13.140:4508 , piece_id=L6UQ6RCDKLFYH35IEPYNSDS5VCSIDAQFGY3EPV4UJ5UFCCRVEOXQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.466266923 (2h59m32.247200311s) [noise=true , address=77.90.31.10:4530 , piece_id=DN6ZKXVXPYWSHH2Q3OMNLI3N5GIPIRE2ZFSXSBLCTSHMSZ223VSQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.466495946 (2h59m32.247429324s) [noise=true , address=104.36.82.54:4157 , piece_id=AVIHP2IPPTNPK2BGGD7DKNN4BC5723UMFX7ZA2Q7XTKGJAPXKMHA, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.466780703 (2h59m32.247714081s) [noise=true , address=79.127.201.215:4347 , piece_id=NR6NDEXP7YCR2ZYMP3IU4OOMUMRNYNVEQ7NJBPM2M46RTFC7LOSQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.466788347 (2h59m32.247721732s) [noise=true , address=77.90.31.21:4896 , piece_id=QZY4O2HSTZOD674LJODDQ3PJSJFDSQ5G7RQ6ATXUSPS7HHMW6EYQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.467122105 (2h59m32.248055487s) [noise=true , address=104.36.82.39:4739 , piece_id=ZBGN22WIAZHAQBSHK47WP4ADWS5J4KQCUEMXQKOXMFVPXYWFRXXQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.467331228 (2h59m32.248264648s) [noise=true , address=104.36.82.47:4552 , piece_id=7VP5PFGCNKTUGZRAMS43RFKQ4QINDF4ZWKGXXL5RIPMJHP5EVOUA, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.467413346 (2h59m32.248346725s) [noise=true , address=104.36.82.46:4488 , piece_id=MOW4O2OMMSQERQAJZ5CBCIOFNQXGH7OGEBWV6RRGFLPCFT5QGOMA, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.467704419 (2h59m32.248637803s) [noise=true , address=79.127.219.38:4003 , piece_id=Z4A2OPY77UA7UH27YMJFE55UZRCMSEY35GBWFRK2S4BCT22GNK5Q, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.467965151 (2h59m32.248898535s) [noise=true , address=156.146.43.225:4448 , piece_id=SIENNUYN3ZYB6DXKJ3HKZ425TT7UZ54TYUKS3QUQGTSIVAMSIBNA, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.470534030 (2h59m32.251467406s) [noise=true , address=77.90.31.13:4658 , piece_id=OAIPNL7ZQ3JNBEUSBNA2PKCAOOZT2PNMPEEMNPEKWYBQ7EZJDKOQ, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.470934636 (2h59m32.25186802s) [noise=true , address=79.127.205.235:4691 , piece_id=NGS3ZRD77CVV7ODEMQAINXJ7TRMQDCBOGI6QCZIAXQHQKSSW4DBA, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.471301486 (2h59m32.252234865s) [noise=true , address=77.90.31.17:4787 , piece_id=BY3LUA3BR7PBIICTN64LJ2YLFZXKDSMDK4NZUYR3ZSGTLD5IJIGA, seg_pos={0 0}, upload=2772]: Uploading piece... Jul 27 00:51:16.560062200 (2h59m32.340995566s) [upload=2772]: Next returned nil segment Jul 27 00:51:16.560079304 (2h59m32.341012669s) [upload=2772]: Waiting for error group managing segments... Jul 27 00:51:16.615087668 (2h59m32.396021044s) [noise=true , address=77.90.31.21:4896 , piece_id=QZY4O2HSTZOD674LJODDQ3PJSJFDSQ5G7RQ6ATXUSPS7HHMW6EYQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:16.623992406 (2h59m32.404925774s) [noise=true , address=77.90.31.13:4658 , piece_id=OAIPNL7ZQ3JNBEUSBNA2PKCAOOZT2PNMPEEMNPEKWYBQ7EZJDKOQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:16.684757936 (2h59m32.465691316s) [noise=true , address=77.90.31.10:4530 , piece_id=DN6ZKXVXPYWSHH2Q3OMNLI3N5GIPIRE2ZFSXSBLCTSHMSZ223VSQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:16.710895912 (2h59m32.49182929s) [noise=true , address=79.127.219.38:4003 , piece_id=Z4A2OPY77UA7UH27YMJFE55UZRCMSEY35GBWFRK2S4BCT22GNK5Q, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:17.033378560 (2h59m32.814311951s) [noise=true , address=79.127.205.234:4601 , piece_id=3WPXFNT3NBPBZMNSBGVM53C7EDZSZ4EXN4MNQNLPDVVOUCTEAFJQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:17.054440292 (2h59m32.835373676s) [noise=true , address=79.127.205.235:4691 , piece_id=NGS3ZRD77CVV7ODEMQAINXJ7TRMQDCBOGI6QCZIAXQHQKSSW4DBA, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:17.093593916 (2h59m32.874527278s) [noise=true , address=79.127.205.233:4656 , piece_id=KCYAWSPDXNB7UIVGLARV2FTM3LBCV2VDRRPNMKJTITT2K2BK7EUQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:17.124755299 (2h59m32.905688736s) [noise=true , address=64.49.13.140:4508 , piece_id=L6UQ6RCDKLFYH35IEPYNSDS5VCSIDAQFGY3EPV4UJ5UFCCRVEOXQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:18.360490287 (2h59m34.141423657s) [noise=true , address=64.49.13.138:4483 , piece_id=SSQZHGTPJK527YBIDR26IR3FQPXLCR5MLIM2KEHT64QBAPWVF2QQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:51:20.186624155 (2h59m35.967557529s) [noise=true , address=156.146.43.225:4448 , piece_id=SIENNUYN3ZYB6DXKJ3HKZ425TT7UZ54TYUKS3QUQGTSIVAMSIBNA, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:25.216256853 (2h59m40.99719022s) [noise=true , address=79.127.201.215:4347 , piece_id=NR6NDEXP7YCR2ZYMP3IU4OOMUMRNYNVEQ7NJBPM2M46RTFC7LOSQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:51:28.049208153 (2h59m43.830141528s) [noise=true , address=79.127.201.215:4356 , piece_id=7FN5YUCYCWM4YZWFZNB3DCW66445K5G4N4XTXACTUD5FKIM6O7VQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: <nil> Jul 27 00:52:16.372095834 (3h0m32.153029221s) [noise=true , address=104.36.82.10:4109 , piece_id=Y43CNHUPXVHVKZYOVOINYZZJYCH4MATGSPJX24VTCESSVTK62BZQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: ecclient: upload failed (node:12ZE1ut9Tum4tzq2m8zbFaGUNLdNZSZQYARkayDAfE1b7uZtWjW, address:104.36.82.10:4109): protocol: rpc: dial tcp 104.36.82.10:4109: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:16.372196170 (3h0m32.153129555s) [noise=true , address=104.36.82.46:4488 , piece_id=MOW4O2OMMSQERQAJZ5CBCIOFNQXGH7OGEBWV6RRGFLPCFT5QGOMA, seg_pos={0 0}, upload=2772]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: rpc: dial tcp 104.36.82.46:4488: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:16.372286528 (3h0m32.153219913s) [noise=true , address=77.90.31.17:4787 , piece_id=BY3LUA3BR7PBIICTN64LJ2YLFZXKDSMDK4NZUYR3ZSGTLD5IJIGA, seg_pos={0 0}, upload=2772]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 159.223.143.39:45536->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:16.372426527 (3h0m32.153359905s) [noise=true , address=104.36.82.47:4552 , piece_id=7VP5PFGCNKTUGZRAMS43RFKQ4QINDF4ZWKGXXL5RIPMJHP5EVOUA, seg_pos={0 0}, upload=2772]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: write tcp 159.223.143.39:45568->104.36.82.47:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:16.372497880 (3h0m32.153431243s) [noise=true , address=104.36.82.39:4739 , piece_id=ZBGN22WIAZHAQBSHK47WP4ADWS5J4KQCUEMXQKOXMFVPXYWFRXXQ, seg_pos={0 0}, upload=2772]: Done uploading piece. err: ecclient: upload failed (node:12A4nWQsS43tSv3atCnj6CB2ZGKFxQUHJp8a7ftP9y1XumeFk33, address:104.36.82.39:4739): protocol: write tcp 159.223.143.39:33296->104.36.82.39:4739: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:16.372510741 (3h0m32.153444111s) [noise=true , address=104.36.82.54:4157 , piece_id=AVIHP2IPPTNPK2BGGD7DKNN4BC5723UMFX7ZA2Q7XTKGJAPXKMHA, seg_pos={0 0}, upload=2772]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:52:16.372530339 (3h0m32.153463716s) [seg_pos={0 0}, upload=2772]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:52:16.399416597 (3h0m32.180349966s) [upload=2772]: Done uploading segments... Jul 27 00:52:16.399441059 (3h0m32.180374434s) [upload=2772]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.029888558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......................X......X...] duration: 212.850591ms FAIL (2025-07-27 00:52:10 UTC) orbiter warning: uplink: too many requests Duration: 212.890448ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X..] duration: 516.449453ms FAIL (2025-07-27 00:52:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 516.481123ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................X...............X...X...............X.] duration: 38.937044229s FAIL (2025-07-27 00:51:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.937065305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X.] duration: 476.173753ms FAIL (2025-07-27 00:51:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 476.210027ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XXX] duration: 30.034368293s FAIL (2025-07-27 00:51:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 49 (2h59m3.529248833s) [noise=true , address=104.36.82.5:4837 , piece_id=YYBUOOGEPDWAJQ7M2T4JWSD5GHZ33TBB2FB6W6NFGR4FTQJMEQFA, seg_pos={0 0}, upload=2757]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:56448->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:47.748344871 (2h59m3.529278258s) [seg_pos={0 0}, upload=2757]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:50:47.787434458 (2h59m3.568367824s) [upload=2757]: Done uploading segments... Jul 27 00:50:47.787467381 (2h59m3.568400764s) [upload=2757]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:51:07.070865304 (2h59m22.851798685s) [upload=2771]: Starting upload Jul 27 00:51:07.070968179 (2h59m22.851901568s) [upload=2771]: Done starting upload Jul 27 00:51:07.071250461 (2h59m22.852183836s) [upload=2771]: Uploading segments... Jul 27 00:51:07.071587536 (2h59m22.852520919s) [upload=2771]: Got next segment. Inline: false Jul 27 00:51:07.071673557 (2h59m22.852606942s) [upload=2771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:51:07.071680889 (2h59m22.852614267s) [upload=2771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:51:07.100062929 (2h59m22.880996317s) [seg_pos={0 0}, upload=2771]: Begin upload segment... Jul 27 00:51:07.100172769 (2h59m22.881106138s) [seg_pos={0 0}, upload=2771]: Done begin upload segment. Jul 27 00:51:07.100211062 (2h59m22.881144448s) [noise=true , address=45.91.116.5:4298 , piece_id=QA3LEEILCENBZM2STYWNO3MWE74XTB4HGEG6GE2KU6Y4Q76EBYYA, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.100852441 (2h59m22.881785828s) [noise=true , address=104.36.82.6:4873 , piece_id=GX6EDTLIYWO3NLOJTSJTLKTGW37W53NXMN72VGDCQ5QPJX24TQAA, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.101304760 (2h59m22.882238146s) [noise=true , address=77.90.31.16:4735 , piece_id=QIKJPFWO4QX3HW4JABGXMVI3FVTMAQBV6UTIXIE42IUN7ZP3VYPQ, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.101827591 (2h59m22.882760974s) [noise=true , address=45.91.116.13:4154 , piece_id=EVWVGOBTAGUZLIINUCA4RFQD2SSTRSYSLHPTPUOMBJ2APBIEI2CA, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.102240128 (2h59m22.883173518s) [noise=true , address=45.91.116.6:4974 , piece_id=VISCU5XYIMXWBAFBSUVLDQAFE5KSKWEF76W4II4G5AHHKJFBAJQQ, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.102686991 (2h59m22.883620381s) [noise=true , address=104.36.82.9:4006 , piece_id=H3LIDOLY6KJA7MI67D7JTEQILSW6PPZJOGVIOQZEQ2N74ZT7K2JQ, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.103223916 (2h59m22.884157302s) [noise=true , address=45.91.116.8:4048 , piece_id=VJWW4MGYXCD4BKVR3MXUVTZACIWZB4Y3CZ5ZROGE3C5UOWJUW4NA, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.103706992 (2h59m22.88464038s) [noise=true , address=45.91.116.12:4225 , piece_id=WUASLDKVLBPKIHBTW3TF5KKUK3HIBT52RZUECEIXVXTRHM5JEQYQ, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.104145723 (2h59m22.885079119s) [noise=true , address=104.36.82.10:4077 , piece_id=U7OAY7BWDMBG43GJZGT53Q3F6P2NRYOFCN4UEUGBEVZ72W3WD5ZA, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.104576118 (2h59m22.885509508s) [noise=true , address=77.90.31.14:4661 , piece_id=UPTY2RWOO7JYM3WPBWXG6QJEUJ7LJYG4D7CNYGVBBZ3YS2LNPV7Q, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.105028725 (2h59m22.885962109s) [noise=true , address=45.91.116.14:4191 , piece_id=GPIOOJCDQVJAAL6KNH6A2GBJZ62A54JHKCXMQFKB3RNNW3I5TGHA, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.105430217 (2h59m22.886363608s) [noise=true , address=45.91.116.4:4940 , piece_id=S73JCTSIRJ3DL7DR3T3ZOKGW5XMOE3NSYGUJH3ENZOZKORIONLXQ, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.105891876 (2h59m22.886825262s) [noise=true , address=45.91.116.10:4083 , piece_id=24RBYY7B6ZAQQRXME6AZN6KKKBHEAZOELNFMF7AQXOECPOSHPF2A, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.106354487 (2h59m22.887287874s) [noise=true , address=104.36.82.8:4971 , piece_id=TGJNCYN2DHEAFWACUIJUAOQ44F737Q3EYMVV6OZ3WS3XF5TFXDSA, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.106749484 (2h59m22.887682871s) [noise=true , address=45.91.116.4:4937 , piece_id=E3JFDGZ4BIT3QD4NY22RZKJSH4PXZC5FEDRXZ2RG6YOKCIMF3VTQ, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.111732567 (2h59m22.892665952s) [noise=true , address=77.90.31.17:4772 , piece_id=WLDXCZIC56WGHEYU5OIMRDJFWXAXBUZZDIT2P2IAQFN6AHQUXJGA, seg_pos={0 0}, upload=2771]: Uploading piece... Jul 27 00:51:07.184278332 (2h59m22.965211695s) [upload=2771]: Next returned nil segment Jul 27 00:51:07.184300527 (2h59m22.965233912s) [upload=2771]: Waiting for error group managing segments... Jul 27 00:51:07.189844189 (2h59m22.970777556s) [noise=true , address=45.91.116.6:4974 , piece_id=VISCU5XYIMXWBAFBSUVLDQAFE5KSKWEF76W4II4G5AHHKJFBAJQQ, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.190219469 (2h59m22.971152834s) [noise=true , address=45.91.116.10:4083 , piece_id=24RBYY7B6ZAQQRXME6AZN6KKKBHEAZOELNFMF7AQXOECPOSHPF2A, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.190388655 (2h59m22.97132202s) [noise=true , address=45.91.116.13:4154 , piece_id=EVWVGOBTAGUZLIINUCA4RFQD2SSTRSYSLHPTPUOMBJ2APBIEI2CA, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.191085673 (2h59m22.972019037s) [noise=true , address=45.91.116.12:4225 , piece_id=WUASLDKVLBPKIHBTW3TF5KKUK3HIBT52RZUECEIXVXTRHM5JEQYQ, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.191654506 (2h59m22.972587887s) [noise=true , address=45.91.116.8:4048 , piece_id=VJWW4MGYXCD4BKVR3MXUVTZACIWZB4Y3CZ5ZROGE3C5UOWJUW4NA, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.192638869 (2h59m22.973572231s) [noise=true , address=45.91.116.4:4937 , piece_id=E3JFDGZ4BIT3QD4NY22RZKJSH4PXZC5FEDRXZ2RG6YOKCIMF3VTQ, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.193168091 (2h59m22.97410146s) [noise=true , address=45.91.116.14:4191 , piece_id=GPIOOJCDQVJAAL6KNH6A2GBJZ62A54JHKCXMQFKB3RNNW3I5TGHA, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.193576977 (2h59m22.974510349s) [noise=true , address=77.90.31.14:4661 , piece_id=UPTY2RWOO7JYM3WPBWXG6QJEUJ7LJYG4D7CNYGVBBZ3YS2LNPV7Q, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.194046843 (2h59m22.974980222s) [noise=true , address=45.91.116.4:4940 , piece_id=S73JCTSIRJ3DL7DR3T3ZOKGW5XMOE3NSYGUJH3ENZOZKORIONLXQ, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:07.194438715 (2h59m22.97537208s) [noise=true , address=45.91.116.5:4298 , piece_id=QA3LEEILCENBZM2STYWNO3MWE74XTB4HGEG6GE2KU6Y4Q76EBYYA, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:08.605669107 (2h59m24.386602489s) [noise=true , address=77.90.31.16:4735 , piece_id=QIKJPFWO4QX3HW4JABGXMVI3FVTMAQBV6UTIXIE42IUN7ZP3VYPQ, seg_pos={0 0}, upload=2771]: Done uploading piece. err: <nil> Jul 27 00:51:37.024915198 (2h59m52.805848574s) [noise=true , address=104.36.82.8:4971 , piece_id=TGJNCYN2DHEAFWACUIJUAOQ44F737Q3EYMVV6OZ3WS3XF5TFXDSA, seg_pos={0 0}, upload=2771]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:37.025878596 (2h59m52.806811959s) [noise=true , address=77.90.31.17:4772 , piece_id=WLDXCZIC56WGHEYU5OIMRDJFWXAXBUZZDIT2P2IAQFN6AHQUXJGA, seg_pos={0 0}, upload=2771]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:37.026021172 (2h59m52.806954534s) [noise=true , address=104.36.82.6:4873 , piece_id=GX6EDTLIYWO3NLOJTSJTLKTGW37W53NXMN72VGDCQ5QPJX24TQAA, seg_pos={0 0}, upload=2771]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:37.026122896 (2h59m52.807056259s) [noise=true , address=104.36.82.10:4077 , piece_id=U7OAY7BWDMBG43GJZGT53Q3F6P2NRYOFCN4UEUGBEVZ72W3WD5ZA, seg_pos={0 0}, upload=2771]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:37.026324264 (2h59m52.807257642s) [noise=true , address=104.36.82.9:4006 , piece_id=H3LIDOLY6KJA7MI67D7JTEQILSW6PPZJOGVIOQZEQ2N74ZT7K2JQ, seg_pos={0 0}, upload=2771]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56302->104.36.82.9:4006: use of closed network connection; context deadline exceeded Jul 27 00:51:37.026349488 (2h59m52.807282875s) [seg_pos={0 0}, upload=2771]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:51:37.055915411 (2h59m52.836848784s) [upload=2771]: Done uploading segments... Jul 27 00:51:37.056209483 (2h59m52.83714286s) [upload=2771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.034396941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......................X......X..] duration: 464.964744ms FAIL (2025-07-27 00:51:30 UTC) orbiter warning: uplink: too many requests Duration: 464.996815ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.................XX....X......XX.XXXX...X.................] duration: 445.742712ms FAIL (2025-07-27 00:51:29 UTC) orbiter warning: context deadline exceeded Duration: 445.833671ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........X] duration: 1m0.02875617s FAIL (2025-07-27 00:51:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== l 27 00:50:16.348453839 (2h58m32.129387222s) [noise=true , address=45.91.116.4:4964 , piece_id=X5XNGUWB33DLARYYRUMP55IMMW7FU5JKIYDEFOWDPDPC5OMPRNOA, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.348716703 (2h58m32.129650089s) [noise=true , address=79.127.205.245:4239 , piece_id=IE7QUX5BNV4UXBUG5CZEDC6EYOAOJTA7RA6VJF45X6JSIGP5Q52Q, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.348829274 (2h58m32.129762662s) [noise=true , address=104.36.82.46:4481 , piece_id=YZP4LF55WCBIPNJZNRJB22EIZLT3T5NMWBMNPOQBP4YFUQMML3HQ, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.349261961 (2h58m32.130195348s) [noise=true , address=104.36.82.52:4090 , piece_id=6I3FRFCUPQS4ICEX4BTAYNLNRSPGBYKA26HMD7XYAOVPDVSBKXJA, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.349346198 (2h58m32.130279584s) [noise=true , address=104.36.82.20:4448 , piece_id=62LLWJIMZR3UO7LDVXCQBLQESJG3RALZUVMUZVYT3EEASNGEPKVQ, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.351227973 (2h58m32.132161363s) [noise=true , address=104.36.82.47:4545 , piece_id=O3OOTFK2K7JDPPMRURT22EMWXLR63W6QTN63UL2YZNBD6TUS3S6Q, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.351681339 (2h58m32.132614721s) [noise=true , address=77.90.31.13:4636 , piece_id=DMTI7FZ3DESXJF5DPYN2KRWI4CDUBSS45KFN43EM6TAEZZ4BXS6A, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.353771166 (2h58m32.134704554s) [noise=true , address=77.90.31.18:4823 , piece_id=SCCBZTDBRH5XQO26AHWIMH4TUK647LSVYNC6ID7YE57BT37X4QYQ, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.354321924 (2h58m32.135255311s) [noise=true , address=79.127.219.38:4001 , piece_id=HXZUKZ7WBK4XCS4GCKGONERVKBR2BNKVCURPIT5KSCT27MNYCPRQ, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.354750989 (2h58m32.135684379s) [noise=true , address=104.36.82.40:4943 , piece_id=UQCAET6CJ7OV6MFFU56IXUF3V6HZGIVGH7BYSPQI2FW3DL34WMFA, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.355317777 (2h58m32.136251164s) [noise=true , address=77.90.31.19:4846 , piece_id=HUZMKTJ5CTJCHUWGHFTHMOXWFHO7JA4AOOMDST2RN2MVIRLDOSQA, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.357394246 (2h58m32.138327635s) [noise=true , address=77.90.31.12:4611 , piece_id=BREOTUQ5IKQ3QYKDNX4U5KSWETPXL5ZT2ZT3WBGHUUQEJ4DJ4C3A, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.358809934 (2h58m32.139743321s) [noise=true , address=104.36.82.21:4497 , piece_id=4CSSL7DF4SLU7ZO426I526B6EPWKVKZJRGB3YBCTMJXGHETXTM4Q, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.359771604 (2h58m32.140704993s) [noise=true , address=77.90.31.16:4748 , piece_id=3IBDTS466TV6RXOEYL5A4PCMAAZMTZ3PQHWUWJ5BGXM2ZBQYD4NA, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.360037041 (2h58m32.140970425s) [noise=true , address=104.36.82.5:4846 , piece_id=PQTC7A47S5KXADRB55N5MNQGHEA4QMVPFNIQKMHQMONXDMLKYIFQ, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.360493770 (2h58m32.141427154s) [noise=true , address=64.49.13.137:4502 , piece_id=3N4Z4FWVS6GK3M3NEDKPUY2NDB23HHT6HXXLON7CZPTWQZIVMH5Q, seg_pos={0 0}, upload=2756]: Uploading piece... Jul 27 00:50:16.416849044 (2h58m32.197782424s) [noise=true , address=77.90.31.16:4748 , piece_id=3IBDTS466TV6RXOEYL5A4PCMAAZMTZ3PQHWUWJ5BGXM2ZBQYD4NA, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: storage node overloaded, request limit: 2000 Jul 27 00:50:16.420487323 (2h58m32.201420725s) [noise=true , address=77.90.31.19:4846 , piece_id=HUZMKTJ5CTJCHUWGHFTHMOXWFHO7JA4AOOMDST2RN2MVIRLDOSQA, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: storage node overloaded, request limit: 2000 Jul 27 00:50:16.495636867 (2h58m32.276570245s) [upload=2756]: Next returned nil segment Jul 27 00:50:16.495658500 (2h58m32.276591883s) [upload=2756]: Waiting for error group managing segments... Jul 27 00:50:16.502288017 (2h58m32.283221385s) [noise=true , address=79.127.219.38:4001 , piece_id=HXZUKZ7WBK4XCS4GCKGONERVKBR2BNKVCURPIT5KSCT27MNYCPRQ, seg_pos={0 0}, upload=2756]: Done uploading piece. err: <nil> Jul 27 00:50:16.502917160 (2h58m32.283850528s) [noise=true , address=77.90.31.13:4636 , piece_id=DMTI7FZ3DESXJF5DPYN2KRWI4CDUBSS45KFN43EM6TAEZZ4BXS6A, seg_pos={0 0}, upload=2756]: Done uploading piece. err: <nil> Jul 27 00:50:16.525330377 (2h58m32.306263745s) [noise=true , address=156.146.43.225:4448 , piece_id=UMQFOJFY65XK7VYO7ND7D35EUROUSFOXC4WXLDMZ2HJV7CMXTSWQ, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:50:16.957925879 (2h58m32.738859261s) [noise=true , address=79.127.205.245:4239 , piece_id=IE7QUX5BNV4UXBUG5CZEDC6EYOAOJTA7RA6VJF45X6JSIGP5Q52Q, seg_pos={0 0}, upload=2756]: Done uploading piece. err: <nil> Jul 27 00:50:16.961615763 (2h58m32.742549125s) [noise=true , address=79.127.205.246:4260 , piece_id=ZHWPPNNUJQUPV3QKKZZFWLMTIDDUHERY7Q42NTNJWB6W4S7BWRVA, seg_pos={0 0}, upload=2756]: Done uploading piece. err: <nil> Jul 27 00:50:17.094955980 (2h58m32.875889358s) [noise=true , address=64.49.13.137:4502 , piece_id=3N4Z4FWVS6GK3M3NEDKPUY2NDB23HHT6HXXLON7CZPTWQZIVMH5Q, seg_pos={0 0}, upload=2756]: Done uploading piece. err: <nil> Jul 27 00:50:20.581204645 (2h58m36.362138024s) [noise=true , address=77.90.31.12:4611 , piece_id=BREOTUQ5IKQ3QYKDNX4U5KSWETPXL5ZT2ZT3WBGHUUQEJ4DJ4C3A, seg_pos={0 0}, upload=2756]: Done uploading piece. err: <nil> Jul 27 00:50:22.941045027 (2h58m38.721978408s) [noise=true , address=45.91.116.4:4964 , piece_id=X5XNGUWB33DLARYYRUMP55IMMW7FU5JKIYDEFOWDPDPC5OMPRNOA, seg_pos={0 0}, upload=2756]: Done uploading piece. err: <nil> Jul 27 00:50:41.764874563 (2h58m57.545807929s) [noise=true , address=77.90.31.18:4823 , piece_id=SCCBZTDBRH5XQO26AHWIMH4TUK647LSVYNC6ID7YE57BT37X4QYQ, seg_pos={0 0}, upload=2756]: Done uploading piece. err: <nil> Jul 27 00:51:16.270361741 (2h59m32.051295124s) [noise=true , address=104.36.82.47:4545 , piece_id=O3OOTFK2K7JDPPMRURT22EMWXLR63W6QTN63UL2YZNBD6TUS3S6Q, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: write tcp 159.223.143.39:57996->104.36.82.47:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:16.270447199 (2h59m32.051380575s) [noise=true , address=104.36.82.46:4481 , piece_id=YZP4LF55WCBIPNJZNRJB22EIZLT3T5NMWBMNPOQBP4YFUQMML3HQ, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:16.270513868 (2h59m32.051447243s) [noise=true , address=104.36.82.21:4497 , piece_id=4CSSL7DF4SLU7ZO426I526B6EPWKVKZJRGB3YBCTMJXGHETXTM4Q, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:104.36.82.21:4497): protocol: write tcp 159.223.143.39:58216->104.36.82.21:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:16.270591397 (2h59m32.051524774s) [noise=true , address=104.36.82.5:4846 , piece_id=PQTC7A47S5KXADRB55N5MNQGHEA4QMVPFNIQKMHQMONXDMLKYIFQ, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: write tcp 159.223.143.39:39506->104.36.82.5:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:16.270610358 (2h59m32.05154372s) [noise=true , address=104.36.82.52:4090 , piece_id=6I3FRFCUPQS4ICEX4BTAYNLNRSPGBYKA26HMD7XYAOVPDVSBKXJA, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 159.223.143.39:49446->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:16.270649570 (2h59m32.051582935s) [noise=true , address=104.36.82.20:4448 , piece_id=62LLWJIMZR3UO7LDVXCQBLQESJG3RALZUVMUZVYT3EEASNGEPKVQ, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp, address:104.36.82.20:4448): protocol: write tcp 159.223.143.39:45904->104.36.82.20:4448: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:16.270685484 (2h59m32.05161886s) [noise=true , address=104.36.82.40:4943 , piece_id=UQCAET6CJ7OV6MFFU56IXUF3V6HZGIVGH7BYSPQI2FW3DL34WMFA, seg_pos={0 0}, upload=2756]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:104.36.82.40:4943): protocol: write tcp 159.223.143.39:59108->104.36.82.40:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:16.270733259 (2h59m32.051666624s) [seg_pos={0 0}, upload=2756]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:51:16.297181531 (2h59m32.078114897s) [upload=2756]: Done uploading segments... Jul 27 00:51:16.297211577 (2h59m32.078144944s) [upload=2756]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.028783723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................X...............X...X...............X] duration: 1m0.001540197s FAIL (2025-07-27 00:51:13 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001565599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.X] duration: 30.03279934s FAIL (2025-07-27 00:51:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.032935877s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............XX] duration: 30.045527753s FAIL (2025-07-27 00:50:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== cclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:38314->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:17.610992576 (2h58m33.391925962s) [seg_pos={0 0}, upload=2749]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:50:17.648384370 (2h58m33.42931775s) [upload=2749]: Done uploading segments... Jul 27 00:50:17.648419611 (2h58m33.429352996s) [upload=2749]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:50:17.778721238 (2h58m33.559654624s) [upload=2757]: Starting upload Jul 27 00:50:17.778822549 (2h58m33.559755919s) [upload=2757]: Done starting upload Jul 27 00:50:17.778958608 (2h58m33.559891972s) [upload=2757]: Uploading segments... Jul 27 00:50:17.779220974 (2h58m33.560154343s) [upload=2757]: Got next segment. Inline: false Jul 27 00:50:17.779287303 (2h58m33.560220689s) [upload=2757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:50:17.779294975 (2h58m33.560228353s) [upload=2757]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:50:17.808011939 (2h58m33.588945328s) [seg_pos={0 0}, upload=2757]: Begin upload segment... Jul 27 00:50:17.808105391 (2h58m33.58903878s) [seg_pos={0 0}, upload=2757]: Done begin upload segment. Jul 27 00:50:17.808396104 (2h58m33.589329491s) [noise=true , address=77.90.31.16:4733 , piece_id=SW2I6OCIW4KHCM2OTSPX2U5ASBRG2QB7MAEXCT7L4P3VWSK3Y7ZA, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.808455989 (2h58m33.589389375s) [noise=true , address=45.91.116.6:4974 , piece_id=N37OEMSUXMERUDK5ZTAK3SWOBWMYUYTTK7BCEJLTFEPOPV2BX3IA, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.808974873 (2h58m33.589908264s) [noise=true , address=45.91.116.7:4010 , piece_id=VDEKDHDEJTML2VQUD3EMDWSYPE2GHRPGWNDQGPVLR4DLEHL67JEQ, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.812091071 (2h58m33.593024458s) [noise=true , address=77.90.31.11:4554 , piece_id=4FRIUO66H2MS27DJVQNJBMBOLGCDP75OOJ5L72ZEUAAUAXOZ5B5Q, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.812868495 (2h58m33.593801862s) [noise=true , address=45.91.116.14:4190 , piece_id=3FUHP5JMVN4MO5MSUKBIJ5HWCMCVNUVBBWJ6ERHLYTBBFJCC2J5A, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.814139925 (2h58m33.595073313s) [noise=true , address=45.91.116.12:4226 , piece_id=F35T5VGZVQJ7X67C3BESAGVZF5TLCV7JRKFOFBIVZXWDHJAMDYSQ, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.814939228 (2h58m33.595872614s) [noise=true , address=104.36.82.15:4258 , piece_id=BBRPI67VYKD5FRPREYQZDLNMYE6IHDZU5RAGOGWMXLGMXLOWF4XA, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.816400784 (2h58m33.597334173s) [noise=true , address=77.90.31.17:4772 , piece_id=ECMLGWZ5L5GMQFREFJVNY2T7W6RMJ5UEUOMDMN4BMJ7ECHZY7DOQ, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.817197657 (2h58m33.598131045s) [noise=true , address=104.36.82.5:4837 , piece_id=YYBUOOGEPDWAJQ7M2T4JWSD5GHZ33TBB2FB6W6NFGR4FTQJMEQFA, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.818164329 (2h58m33.599097716s) [noise=true , address=45.91.116.13:4153 , piece_id=IIWKOZYL7BA37REPHA4VEQO2XMHY7A4KBGQWDX3BQQEN4HJJY2SQ, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.818943042 (2h58m33.59987643s) [noise=true , address=45.91.116.4:4940 , piece_id=EZIXUWY3HVOWKKLQ5FJVFE5IJFMJHO3UYKPNTLVAX3OOE5EGJSKQ, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.827227463 (2h58m33.60816085s) [noise=true , address=104.36.82.10:4079 , piece_id=PKXWWFXDP2YKDYE3SKKRFUY3VBJN2MRFPBDHXIMMZB67A7MXQTIA, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.831149814 (2h58m33.612083202s) [noise=true , address=77.90.31.14:4661 , piece_id=TD54QZXQ7ZZ4APMVDW53LYAFBQIALYES4ECCM3EJEMV7L5CQGBVQ, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.837823721 (2h58m33.618757107s) [noise=true , address=77.90.31.13:4627 , piece_id=76FTFKTRBVA4JKXPEGGXWUKSQFV6KAS626FPGRGEX7GOVYIV7ZOA, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.841986949 (2h58m33.622920327s) [noise=true , address=45.91.116.14:4189 , piece_id=6GZUBUWPN6FS7LQQOAWMRVADGC5DZRLT62NYX6E2BSQAJMWEF5JQ, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.844460116 (2h58m33.625393496s) [noise=true , address=104.36.82.2:4729 , piece_id=H7IIH6WWWWFFN6PAWLL4ZGIAN4VRYFUELCRRZOHMZ5HXVOM34X6A, seg_pos={0 0}, upload=2757]: Uploading piece... Jul 27 00:50:17.933472316 (2h58m33.714405686s) [upload=2757]: Next returned nil segment Jul 27 00:50:17.933492984 (2h58m33.714426369s) [upload=2757]: Waiting for error group managing segments... Jul 27 00:50:17.940493655 (2h58m33.721427034s) [noise=true , address=45.91.116.12:4226 , piece_id=F35T5VGZVQJ7X67C3BESAGVZF5TLCV7JRKFOFBIVZXWDHJAMDYSQ, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.940854951 (2h58m33.721788322s) [noise=true , address=45.91.116.13:4153 , piece_id=IIWKOZYL7BA37REPHA4VEQO2XMHY7A4KBGQWDX3BQQEN4HJJY2SQ, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.941283383 (2h58m33.722216752s) [noise=true , address=45.91.116.14:4190 , piece_id=3FUHP5JMVN4MO5MSUKBIJ5HWCMCVNUVBBWJ6ERHLYTBBFJCC2J5A, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.941767127 (2h58m33.722700504s) [noise=true , address=77.90.31.11:4554 , piece_id=4FRIUO66H2MS27DJVQNJBMBOLGCDP75OOJ5L72ZEUAAUAXOZ5B5Q, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.941825604 (2h58m33.722758982s) [noise=true , address=77.90.31.13:4627 , piece_id=76FTFKTRBVA4JKXPEGGXWUKSQFV6KAS626FPGRGEX7GOVYIV7ZOA, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.942310979 (2h58m33.723244346s) [noise=true , address=45.91.116.7:4010 , piece_id=VDEKDHDEJTML2VQUD3EMDWSYPE2GHRPGWNDQGPVLR4DLEHL67JEQ, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.942561581 (2h58m33.723494962s) [noise=true , address=77.90.31.14:4661 , piece_id=TD54QZXQ7ZZ4APMVDW53LYAFBQIALYES4ECCM3EJEMV7L5CQGBVQ, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.942983544 (2h58m33.723916916s) [noise=true , address=45.91.116.4:4940 , piece_id=EZIXUWY3HVOWKKLQ5FJVFE5IJFMJHO3UYKPNTLVAX3OOE5EGJSKQ, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.943425541 (2h58m33.724358922s) [noise=true , address=45.91.116.14:4189 , piece_id=6GZUBUWPN6FS7LQQOAWMRVADGC5DZRLT62NYX6E2BSQAJMWEF5JQ, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:17.943618043 (2h58m33.72455141s) [noise=true , address=45.91.116.6:4974 , piece_id=N37OEMSUXMERUDK5ZTAK3SWOBWMYUYTTK7BCEJLTFEPOPV2BX3IA, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:19.433443647 (2h58m35.214377017s) [noise=true , address=77.90.31.16:4733 , piece_id=SW2I6OCIW4KHCM2OTSPX2U5ASBRG2QB7MAEXCT7L4P3VWSK3Y7ZA, seg_pos={0 0}, upload=2757]: Done uploading piece. err: <nil> Jul 27 00:50:47.747214801 (2h59m3.52814818s) [noise=true , address=104.36.82.2:4729 , piece_id=H7IIH6WWWWFFN6PAWLL4ZGIAN4VRYFUELCRRZOHMZ5HXVOM34X6A, seg_pos={0 0}, upload=2757]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:55672->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:47.747517324 (2h59m3.528450725s) [noise=true , address=104.36.82.10:4079 , piece_id=PKXWWFXDP2YKDYE3SKKRFUY3VBJN2MRFPBDHXIMMZB67A7MXQTIA, seg_pos={0 0}, upload=2757]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:41738->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:47.747930970 (2h59m3.528864347s) [noise=true , address=104.36.82.15:4258 , piece_id=BBRPI67VYKD5FRPREYQZDLNMYE6IHDZU5RAGOGWMXLGMXLOWF4XA, seg_pos={0 0}, upload=2757]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:59742->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:47.748223938 (2h59m3.529157318s) [noise=true , address=77.90.31.17:4772 , piece_id=ECMLGWZ5L5GMQFREFJVNY2T7W6RMJ5UEUOMDMN4BMJ7ECHZY7DOQ, seg_pos={0 0}, upload=2757]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 159.223.143.39:48564->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:47.748315449 (2h59m3.529248833s) [noise=true , address=104.36.82.5:4837 , piece_id=YYBUOOGEPDWAJQ7M2T4JWSD5GHZ33TBB2FB6W6NFGR4FTQJMEQFA, seg_pos={0 0}, upload=2757]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:56448->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:47.748344871 (2h59m3.529278258s) [seg_pos={0 0}, upload=2757]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:50:47.787434458 (2h59m3.568367824s) [upload=2757]: Done uploading segments... Jul 27 00:50:47.787467381 (2h59m3.568400764s) [upload=2757]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.045554345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......................X......X.] duration: 218.369532ms FAIL (2025-07-27 00:50:45 UTC) orbiter warning: uplink: too many requests Duration: 218.399806ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.................XX....X......XX.XXXX...X................] duration: 1.823988307s FAIL (2025-07-27 00:50:21 UTC) orbiter warning: context deadline exceeded Duration: 1.824037838s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............X] duration: 30.045828408s FAIL (2025-07-27 00:50:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== n closed: op Jul 27 00:49:11.413386815 (2h57m27.19432019s) [noise=true , address=104.36.82.8:4972 , piece_id=JPGJYCWZTABB2EAFL52PXGYCB566BP3XSGX65NSPLL2QMIRBY35A, seg_pos={0 0}, upload=2740]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 159.223.143.39:50682->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:49:11.413398149 (2h57m27.194331526s) [seg_pos={0 0}, upload=2740]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:49:11.413466235 (2h57m27.194399614s) [upload=2740]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:49:11.413470578 (2h57m27.194403953s) [upload=2740]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:49:11.465748985 (2h57m27.246682354s) [upload=2740]: Done uploading segments... Jul 27 00:49:11.465780608 (2h57m27.246713973s) [upload=2740]: Upload finished. err: <nil> Jul 27 00:49:47.642856397 (2h58m3.423789763s) [upload=2749]: Starting upload Jul 27 00:49:47.642926157 (2h58m3.423859535s) [upload=2749]: Done starting upload Jul 27 00:49:47.643130023 (2h58m3.424063398s) [upload=2749]: Uploading segments... Jul 27 00:49:47.643312319 (2h58m3.424245683s) [upload=2749]: Got next segment. Inline: false Jul 27 00:49:47.643351542 (2h58m3.424284909s) [upload=2749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:49:47.643356402 (2h58m3.424289777s) [upload=2749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:49:47.670584979 (2h58m3.451518355s) [seg_pos={0 0}, upload=2749]: Begin upload segment... Jul 27 00:49:47.670640572 (2h58m3.451573958s) [seg_pos={0 0}, upload=2749]: Done begin upload segment. Jul 27 00:49:47.670726355 (2h58m3.451659739s) [noise=true , address=45.91.116.5:4298 , piece_id=5US225QVKGJ3CRVSZ5YULAIII7BUMOMJWCQNXWHT44UJK6TV6RIQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.670771288 (2h58m3.451704663s) [noise=true , address=77.90.31.21:4877 , piece_id=JUNQVPUSQSU76NK7KQIBYKYMKR5VUZTRAQGBW5ZYZF7XRV7SJNOA, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.671660832 (2h58m3.452594218s) [noise=true , address=104.36.82.14:4221 , piece_id=ZGARBTKHQQG3JW2H46YGWZMPY2HOCA4IVBEMFWNNIFN4ZTDSJWKQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.671718156 (2h58m3.452651538s) [noise=true , address=77.90.31.12:4589 , piece_id=OLM6XFPEYINIP632GENP2PVDLF2SLHQWNIJRMLL4OKHVXT7GUOTQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.672595675 (2h58m3.45352906s) [noise=true , address=45.91.116.10:4083 , piece_id=VGLRQ3JUSIWHQA4WQPOEZBZPDVUWNIWLY3HZ7VIKR4LWHJPCPO7Q, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.672780505 (2h58m3.453713889s) [noise=true , address=45.91.116.5:4297 , piece_id=VUCRUK6AUXJP3XDRUAP6PV5YKEOJ3MKR7AQXY3YT4H5NWNI4XSPA, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.673532172 (2h58m3.454465562s) [noise=true , address=45.91.116.7:4011 , piece_id=N75IMVL36GVM676EELRN2UD6ZOBD7PCWHXG3GTBT2GJA674YVYHQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.674848055 (2h58m3.45578144s) [noise=true , address=104.36.82.6:4874 , piece_id=6VA54F64NUHTRGTABBV63M3N5ZTR5DU63PQVG22JZN2A5WDLGHBQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.675302589 (2h58m3.456235975s) [noise=true , address=104.36.82.9:4007 , piece_id=6OQMQ25B2P74LB7JYEH3FWEHXE7ICHQYQGN2AGIZSLIXQBSNQM4A, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.675698904 (2h58m3.456632277s) [noise=true , address=45.91.116.7:4010 , piece_id=ULQWHO6K25LRBHIFTM2E6L32Y4E6CW4BVL7O2LMNOKKOYKCP5FAQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.675712051 (2h58m3.45664543s) [noise=true , address=104.36.82.4:4802 , piece_id=2WWPCD7TIWUK3IV3BQTRNFOM7I6UVCV3C5ZL6RL74OC5NSFXWAKA, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.676084069 (2h58m3.457017454s) [noise=true , address=45.91.116.6:4973 , piece_id=GUP6PEW4CNHSC47QEC6JJIFEWW5VXDJLEGEPDFLKM23P7OMZXOYQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.676633700 (2h58m3.457567073s) [noise=true , address=45.91.116.8:4047 , piece_id=ZKPMNVI7A7BITCYTX6AMPCXUCL7SQC6K4BOZTBWP2M4HKHXJOPEQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.677352475 (2h58m3.458285851s) [noise=true , address=45.91.116.9:4263 , piece_id=ZMJTMJZKFOWSG3NWUPJM2LYJW5SA7UABBG7BY72OS4VFCWK4Q2FA, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.678206748 (2h58m3.459140124s) [noise=true , address=45.91.116.14:4190 , piece_id=DRRPVBZVUD2I3YDZIHIPYQD6ZBGFG5COTZFPXOQDMYFXIIT7LHUA, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.679012687 (2h58m3.459946059s) [noise=true , address=45.91.116.11:4119 , piece_id=RXIISE5TPWZF7253J7ZPUAZXXZLWYVJ4ZE4ZOZBQI6LMTCEFF3JQ, seg_pos={0 0}, upload=2749]: Uploading piece... Jul 27 00:49:47.780035682 (2h58m3.560969061s) [upload=2749]: Next returned nil segment Jul 27 00:49:47.780057639 (2h58m3.560991026s) [upload=2749]: Waiting for error group managing segments... Jul 27 00:49:47.784175957 (2h58m3.565109354s) [noise=true , address=45.91.116.8:4047 , piece_id=ZKPMNVI7A7BITCYTX6AMPCXUCL7SQC6K4BOZTBWP2M4HKHXJOPEQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.785750059 (2h58m3.566683442s) [noise=true , address=45.91.116.6:4973 , piece_id=GUP6PEW4CNHSC47QEC6JJIFEWW5VXDJLEGEPDFLKM23P7OMZXOYQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.787099380 (2h58m3.568032758s) [noise=true , address=45.91.116.11:4119 , piece_id=RXIISE5TPWZF7253J7ZPUAZXXZLWYVJ4ZE4ZOZBQI6LMTCEFF3JQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.788229815 (2h58m3.569163185s) [noise=true , address=45.91.116.14:4190 , piece_id=DRRPVBZVUD2I3YDZIHIPYQD6ZBGFG5COTZFPXOQDMYFXIIT7LHUA, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.788273953 (2h58m3.569207318s) [noise=true , address=45.91.116.9:4263 , piece_id=ZMJTMJZKFOWSG3NWUPJM2LYJW5SA7UABBG7BY72OS4VFCWK4Q2FA, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.788360987 (2h58m3.56929435s) [noise=true , address=45.91.116.5:4298 , piece_id=5US225QVKGJ3CRVSZ5YULAIII7BUMOMJWCQNXWHT44UJK6TV6RIQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.788899877 (2h58m3.569833249s) [noise=true , address=77.90.31.12:4589 , piece_id=OLM6XFPEYINIP632GENP2PVDLF2SLHQWNIJRMLL4OKHVXT7GUOTQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.789100998 (2h58m3.570034381s) [noise=true , address=45.91.116.7:4011 , piece_id=N75IMVL36GVM676EELRN2UD6ZOBD7PCWHXG3GTBT2GJA674YVYHQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.789490458 (2h58m3.570423838s) [noise=true , address=45.91.116.7:4010 , piece_id=ULQWHO6K25LRBHIFTM2E6L32Y4E6CW4BVL7O2LMNOKKOYKCP5FAQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.789918662 (2h58m3.570852045s) [noise=true , address=45.91.116.10:4083 , piece_id=VGLRQ3JUSIWHQA4WQPOEZBZPDVUWNIWLY3HZ7VIKR4LWHJPCPO7Q, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:47.790072446 (2h58m3.571005811s) [noise=true , address=45.91.116.5:4297 , piece_id=VUCRUK6AUXJP3XDRUAP6PV5YKEOJ3MKR7AQXY3YT4H5NWNI4XSPA, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:49:48.122590524 (2h58m3.903523909s) [noise=true , address=77.90.31.21:4877 , piece_id=JUNQVPUSQSU76NK7KQIBYKYMKR5VUZTRAQGBW5ZYZF7XRV7SJNOA, seg_pos={0 0}, upload=2749]: Done uploading piece. err: <nil> Jul 27 00:50:17.610684513 (2h58m33.391617895s) [noise=true , address=104.36.82.14:4221 , piece_id=ZGARBTKHQQG3JW2H46YGWZMPY2HOCA4IVBEMFWNNIFN4ZTDSJWKQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:17.610797492 (2h58m33.391730877s) [noise=true , address=104.36.82.6:4874 , piece_id=6VA54F64NUHTRGTABBV63M3N5ZTR5DU63PQVG22JZN2A5WDLGHBQ, seg_pos={0 0}, upload=2749]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:17.610880135 (2h58m33.391813505s) [noise=true , address=104.36.82.4:4802 , piece_id=2WWPCD7TIWUK3IV3BQTRNFOM7I6UVCV3C5ZL6RL74OC5NSFXWAKA, seg_pos={0 0}, upload=2749]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:17.610957451 (2h58m33.391890822s) [noise=true , address=104.36.82.9:4007 , piece_id=6OQMQ25B2P74LB7JYEH3FWEHXE7ICHQYQGN2AGIZSLIXQBSNQM4A, seg_pos={0 0}, upload=2749]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:38314->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:17.610992576 (2h58m33.391925962s) [seg_pos={0 0}, upload=2749]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:50:17.648384370 (2h58m33.42931775s) [upload=2749]: Done uploading segments... Jul 27 00:50:17.648419611 (2h58m33.429352996s) [upload=2749]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.045871664s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......................X......X] duration: 107.67081ms FAIL (2025-07-27 00:50:17 UTC) orbiter warning: uplink: too many requests Duration: 107.690551ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.......................X......] duration: 222.579003ms FAIL (2025-07-27 00:49:55 UTC) orbiter warning: uplink: invalid range 0:1024 (size:1024) Duration: 222.654346ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.................XX....X......XX.XXXX...X...............] duration: 513.450878ms FAIL (2025-07-27 00:49:32 UTC) orbiter warning: context deadline exceeded Duration: 513.484532ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration: 562.008085ms FAIL (2025-07-27 00:49:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 562.157671ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [X..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X.] duration: 367.082626ms FAIL (2025-07-27 00:49:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 367.119671ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X...............X...X...............] duration: 900.012067ms FAIL (2025-07-27 00:49:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 900.039317ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..............] duration: 404.67629ms FAIL (2025-07-27 00:49:11 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.702514ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.......................X.....] duration: 297.87243ms FAIL (2025-07-27 00:49:03 UTC) orbiter warning: uplink: invalid range 0:1024 (size:1024) Duration: 297.91496ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration: 1.960981231s FAIL (2025-07-27 00:49:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.961014704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X...X.................XX....X......XX.XXXX...X..............] duration: 480.810033ms FAIL (2025-07-27 00:48:52 UTC) orbiter warning: context deadline exceeded Duration: 481.566588ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...............X...X..............] duration: 1.097898099s FAIL (2025-07-27 00:48:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.097972177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration: 26.961483997s FAIL (2025-07-27 00:48:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 26.961514964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [XX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......X] duration: 30.027926075s FAIL (2025-07-27 00:48:47 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027993891s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)