Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 300164 reports (Page 5509 of 6004)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 30.030561403s FAIL (2025-07-26 23:51:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.030592445s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...............X..] duration: 1.889780951s FAIL (2025-07-26 23:51:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.889810249s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033701335s FAIL (2025-07-26 23:50:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== ... Jul 26 23:49:51.062512474 (1h58m6.843445839s) [seg_pos={0 0}, upload=1810]: Done begin upload segment. Jul 26 23:49:51.062567904 (1h58m6.843501287s) [noise=true , address=104.36.82.45:4910 , piece_id=2FPGI2PLU5S6Q5YDWSSDYT35OAKTCFHPUMRPHUMBDUU2PUOEH2YQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.062600660 (1h58m6.843534042s) [noise=true , address=79.127.201.215:4351 , piece_id=ZGV32GT6FLCDUF45JZLKJ7OMXH7RIJ5W7IINGHFTW33VBQEDMMOQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.063098862 (1h58m6.844032243s) [noise=true , address=77.90.31.15:4714 , piece_id=H66HQNWM2QUA36L2TWQBMBV62EJXUIT4BCXFPYRIMWQY3SOCQGZA, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.063249255 (1h58m6.844182645s) [noise=true , address=77.90.31.11:4562 , piece_id=YBE3OL6TWY7ZKD4JKVA3SM3SJTIUOL2FQNBH76L43GR22ZANTZRA, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.063530794 (1h58m6.844464162s) [noise=true , address=79.127.219.38:4015 , piece_id=P6QFOWJINWHOZSHXMRESNEKHVJDC7LKQN7DZBOA3EVO2CVNBUHDQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.063781607 (1h58m6.844714986s) [noise=true , address=79.127.205.234:4609 , piece_id=BU23SDJE7IA5SWDWPQ4YACDNOM667EUJL7WQTPXKNFVKIWBK2PRA, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.063958145 (1h58m6.84489164s) [noise=true , address=104.36.82.46:4482 , piece_id=DZCSV5P6GZZFBY5G6X6GCTCBTWPWO6L47AGJP4F6D2TGCQDEBXVQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.064564164 (1h58m6.845497545s) [noise=true , address=45.91.116.9:4277 , piece_id=EGV63SXH2PZ2VSUWPWHYMRJSCSH65GYQWD3L5NCNCM6EGDXOHXMA, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.065420599 (1h58m6.846353987s) [noise=true , address=104.36.82.55:4224 , piece_id=V76WAOWB6BNQ3ZRBVDFMVCKU3RQE5QMIMMBJWFUWRRDGDXIWJCHQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.066134773 (1h58m6.847068153s) [noise=true , address=64.49.13.154:4927 , piece_id=HYJ622FJCHC3LSMCZS6GGTCCPX6P6UIFGYTVGGSPKRVXJMFDRWYQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.067323522 (1h58m6.848256908s) [noise=true , address=79.127.205.245:4248 , piece_id=HWQLK367ZLE2OXRILUK53PSTJH6CPQ5VGR4L4OK2G7W43JHPFAXQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.067916192 (1h58m6.84884957s) [noise=true , address=104.36.82.5:4870 , piece_id=XBPXX2Y2SSS2EAHRFRN6RCZDFOKIU5ECYSDSSMT6I2WTKOEU7JMA, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.068486830 (1h58m6.849420209s) [noise=true , address=77.90.31.17:4787 , piece_id=GXREXTJ5NGAO3WGC2NECV32J4FLJMWKY662TAK2YWGZDEV5RRPHA, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.068920439 (1h58m6.849853821s) [noise=true , address=77.90.31.16:4755 , piece_id=2DVYODA3WJFOS2J3X6CRY2G7TRVS334B5TT3LWM37JW3XZLTI7CA, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.069381010 (1h58m6.850314395s) [noise=true , address=104.36.82.47:4526 , piece_id=4VPISVBP743IMA7KYBFJHPJ7P4VVQ655KHN3BX7FMMY3DZTB2CGQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.069780164 (1h58m6.850713553s) [noise=true , address=104.36.82.52:4113 , piece_id=TREYS247R3PIIB7ASLO5VONA2HXMAFDWE7MD25X6VBZJ2SAI3QOQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.070273180 (1h58m6.851206569s) [noise=true , address=104.36.82.54:4159 , piece_id=S6CHS7J4KRSHHW3KSCOTM6LQSNQX6D5HG5IRBNKYOFRMOAWO2PBA, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.070718840 (1h58m6.851652223s) [noise=true , address=64.49.13.140:4508 , piece_id=IMWKZIRVHMQXP2QA2T5RHIKIEZHEMPTWKADFRQHVWKIFVC75IKKQ, seg_pos={0 0}, upload=1810]: Uploading piece... Jul 26 23:49:51.151447319 (1h58m6.932380704s) [upload=1810]: Next returned nil segment Jul 26 23:49:51.151466710 (1h58m6.932400107s) [upload=1810]: Waiting for error group managing segments... Jul 26 23:49:51.189817686 (1h58m6.970751063s) [noise=true , address=79.127.219.38:4015 , piece_id=P6QFOWJINWHOZSHXMRESNEKHVJDC7LKQN7DZBOA3EVO2CVNBUHDQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:49:51.194075321 (1h58m6.975008695s) [noise=true , address=77.90.31.15:4714 , piece_id=H66HQNWM2QUA36L2TWQBMBV62EJXUIT4BCXFPYRIMWQY3SOCQGZA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:49:51.280470482 (1h58m7.061403851s) [noise=true , address=77.90.31.11:4562 , piece_id=YBE3OL6TWY7ZKD4JKVA3SM3SJTIUOL2FQNBH76L43GR22ZANTZRA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:49:51.520768266 (1h58m7.301701636s) [noise=true , address=64.49.13.154:4927 , piece_id=HYJ622FJCHC3LSMCZS6GGTCCPX6P6UIFGYTVGGSPKRVXJMFDRWYQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k, address:64.49.13.154:4927): protocol: storage node overloaded, request limit: 2000 Jul 26 23:49:51.664984244 (1h58m7.445917613s) [noise=true , address=79.127.205.245:4248 , piece_id=HWQLK367ZLE2OXRILUK53PSTJH6CPQ5VGR4L4OK2G7W43JHPFAXQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:49:51.682657078 (1h58m7.463590448s) [noise=true , address=79.127.205.234:4609 , piece_id=BU23SDJE7IA5SWDWPQ4YACDNOM667EUJL7WQTPXKNFVKIWBK2PRA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:49:56.803533811 (1h58m12.584467174s) [noise=true , address=77.90.31.17:4787 , piece_id=GXREXTJ5NGAO3WGC2NECV32J4FLJMWKY662TAK2YWGZDEV5RRPHA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:49:58.348839446 (1h58m14.129772827s) [noise=true , address=64.49.13.140:4508 , piece_id=IMWKZIRVHMQXP2QA2T5RHIKIEZHEMPTWKADFRQHVWKIFVC75IKKQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:50:01.403142927 (1h58m17.18407632s) [noise=true , address=45.91.116.9:4277 , piece_id=EGV63SXH2PZ2VSUWPWHYMRJSCSH65GYQWD3L5NCNCM6EGDXOHXMA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:50:25.435534628 (1h58m41.216468005s) [noise=true , address=79.127.201.215:4351 , piece_id=ZGV32GT6FLCDUF45JZLKJ7OMXH7RIJ5W7IINGHFTW33VBQEDMMOQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:50:47.537991050 (1h59m3.318924424s) [noise=true , address=77.90.31.16:4755 , piece_id=2DVYODA3WJFOS2J3X6CRY2G7TRVS334B5TT3LWM37JW3XZLTI7CA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil> Jul 26 23:50:50.996366693 (1h59m6.777300069s) [noise=true , address=104.36.82.47:4526 , piece_id=4VPISVBP743IMA7KYBFJHPJ7P4VVQ655KHN3BX7FMMY3DZTB2CGQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: rpc: dial tcp 104.36.82.47:4526: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:50.996541596 (1h59m6.777474978s) [noise=true , address=104.36.82.55:4224 , piece_id=V76WAOWB6BNQ3ZRBVDFMVCKU3RQE5QMIMMBJWFUWRRDGDXIWJCHQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:43862->104.36.82.55:4224: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:50.996618248 (1h59m6.77755163s) [noise=true , address=104.36.82.52:4113 , piece_id=TREYS247R3PIIB7ASLO5VONA2HXMAFDWE7MD25X6VBZJ2SAI3QOQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou, address:104.36.82.52:4113): protocol: write tcp 159.223.143.39:59156->104.36.82.52:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:50.996658963 (1h59m6.777592341s) [noise=true , address=104.36.82.5:4870 , piece_id=XBPXX2Y2SSS2EAHRFRN6RCZDFOKIU5ECYSDSSMT6I2WTKOEU7JMA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:1MmdhnDp6kKktFb5btZUDeVQ2u3CNUKCZU7RfuEqiSQ2TEqwKX, address:104.36.82.5:4870): protocol: rpc: dial tcp 104.36.82.5:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:50.996728285 (1h59m6.777661664s) [noise=true , address=104.36.82.45:4910 , piece_id=2FPGI2PLU5S6Q5YDWSSDYT35OAKTCFHPUMRPHUMBDUU2PUOEH2YQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12bRYzNggkq7QWFTP9fB94dTrMMX8AYvKMt2zjmJBzT1VwqyPVZ, address:104.36.82.45:4910): protocol: rpc: dial tcp 104.36.82.45:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:50.996766567 (1h59m6.777699929s) [noise=true , address=104.36.82.54:4159 , piece_id=S6CHS7J4KRSHHW3KSCOTM6LQSNQX6D5HG5IRBNKYOFRMOAWO2PBA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:104.36.82.54:4159): protocol: rpc: dial tcp 104.36.82.54:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:50.996780048 (1h59m6.777713411s) [noise=true , address=104.36.82.46:4482 , piece_id=DZCSV5P6GZZFBY5G6X6GCTCBTWPWO6L47AGJP4F6D2TGCQDEBXVQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:50.996803866 (1h59m6.777737256s) [seg_pos={0 0}, upload=1810]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:50:51.027692779 (1h59m6.808626147s) [upload=1810]: Done uploading segments... Jul 26 23:50:51.027740654 (1h59m6.808674035s) [upload=1810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.033726328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...............X.] duration: 27.641707061s FAIL (2025-07-26 23:50:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.641799369s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X...X..XX.X...X..X.......X...X....XX..............X.X.......] duration: 35.104111093s FAIL (2025-07-26 23:50:38 UTC) context deadline exceeded Duration: 35.10414243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 54 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.028116557s FAIL (2025-07-26 23:50:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.02814762s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 30.028644475s FAIL (2025-07-26 23:50:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 15.718809995s) [noise=true , address=104.36.82.6:4874 , piece_id=GQDCX4I2PWKO76IAUPFMKZ6P34CUCHAFKP6VG2DZTV2H33M2KATQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:59.937906032 (1h57m15.718839395s) [seg_pos={0 0}, upload=1791]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:48:59.961543129 (1h57m15.742476505s) [upload=1791]: Done uploading segments... Jul 26 23:48:59.961589585 (1h57m15.742522967s) [upload=1791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 23:49:44.783843222 (1h58m0.564776602s) [upload=1806]: Starting upload Jul 26 23:49:44.783927151 (1h58m0.564860514s) [upload=1806]: Done starting upload Jul 26 23:49:44.784027248 (1h58m0.564960628s) [upload=1806]: Uploading segments... Jul 26 23:49:44.784417545 (1h58m0.565350923s) [upload=1806]: Got next segment. Inline: false Jul 26 23:49:44.784467247 (1h58m0.565400624s) [upload=1806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:49:44.784475657 (1h58m0.565409037s) [upload=1806]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:49:44.828827465 (1h58m0.609760831s) [seg_pos={0 0}, upload=1806]: Begin upload segment... Jul 26 23:49:44.828899983 (1h58m0.609833352s) [seg_pos={0 0}, upload=1806]: Done begin upload segment. Jul 26 23:49:44.828957355 (1h58m0.609890742s) [noise=true , address=77.90.31.14:4662 , piece_id=XYGW55XYY3Q5UP3TWQAA6GFZDVSGKOQOK44RRLZNBM2ODSMXKK4Q, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.829492531 (1h58m0.610425918s) [noise=true , address=104.36.82.15:4259 , piece_id=IP2ME27ARN24X6SQIWJ4VJZO3DRT3QR5X62D3PXAGPKJSZLJJBJQ, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.829932411 (1h58m0.610865795s) [noise=true , address=45.91.116.7:4010 , piece_id=FPSXDJGHNIJF6RHZ4TPIW3B4O6VT3VHE2RFECVTJCTBJVT57Y52A, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.830521335 (1h58m0.611454714s) [noise=true , address=77.90.31.12:4592 , piece_id=UFA3FAYUMNPK6W54LHM3S5CL5L7DD5GOABZ6XRGEY4WLQMSJIELQ, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.830943966 (1h58m0.611877345s) [noise=true , address=104.36.82.11:4113 , piece_id=O3OLP35I72UTEBMZUCUDLKPMYCZ4VN7AQNP7W2Z5DPS3UCL4EZLQ, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.831135649 (1h58m0.612069031s) [noise=true , address=45.91.116.9:4262 , piece_id=K5NV5HLMNZFHB7PEOE2STOFG44GB3DS7V56BTOIHFXPY2MYREWBA, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.831364431 (1h58m0.612297812s) [noise=true , address=77.90.31.21:4879 , piece_id=VVX5RS3HRCFW46VDPA6EWS5XPX4VNRPNEO4OA6ZOOXIACC5Y7GUQ, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.831667623 (1h58m0.612600997s) [noise=true , address=77.90.31.11:4553 , piece_id=P65FDAFZYLPPATJ5JNP7KQ4MVOR7K4HIVZVDZPRI4I32NFIETDWQ, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.831770441 (1h58m0.612703828s) [noise=true , address=77.90.31.15:4698 , piece_id=BJ36WSTZOGQWMIDB6NNGPSS66J5QA7VEQM5ZIVGCKRPRR6OXYT2Q, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.832218793 (1h58m0.613152189s) [noise=true , address=104.36.82.5:4839 , piece_id=FOFV7KSLZQRGIYF34ROV7QWSN2NZTUVHP3CKKGTBIPKRIQ3T3PCA, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.832294596 (1h58m0.613227973s) [noise=true , address=45.91.116.13:4155 , piece_id=UHF77UOL55YIPUQROVVXOOEMRBKMTMOCOGQZSHDUYXUKSUVQBFSQ, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.832609944 (1h58m0.613543321s) [noise=true , address=104.36.82.15:4257 , piece_id=FYEG6JRAG6GX4OC3TJO553FLTXVX4JUHJU4VAFZHYCCOKOGA5QLA, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.832988392 (1h58m0.613921772s) [noise=true , address=45.91.116.8:4048 , piece_id=PCNJHK3FGP2PO2DSHSTIHJ6PGR4ERWS6XW3WTWWIURBYGX4RYF3A, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.833364868 (1h58m0.614298252s) [noise=true , address=104.36.82.6:4875 , piece_id=4GPNO2DZZ2HLNE4DGZVT2AKYTYP5ODGF7VIXAXPFBN7OGTVKM5TQ, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.833711978 (1h58m0.614645355s) [noise=true , address=77.90.31.14:4661 , piece_id=GJGHG6U6KNCISVHE6R74LZV6FKUO65T4O6ENJSFOEIDQ6QU2VLAA, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.834092083 (1h58m0.615025466s) [noise=true , address=45.91.116.6:4973 , piece_id=LOUDCZH4QNZ6HNKFPWOGD3DHJCG45L5LRTT3A2CN6BOVE5OI7SEA, seg_pos={0 0}, upload=1806]: Uploading piece... Jul 26 23:49:44.929003034 (1h58m0.709936406s) [upload=1806]: Next returned nil segment Jul 26 23:49:44.929023757 (1h58m0.709957121s) [upload=1806]: Waiting for error group managing segments... Jul 26 23:49:44.933782221 (1h58m0.714715583s) [noise=true , address=45.91.116.6:4973 , piece_id=LOUDCZH4QNZ6HNKFPWOGD3DHJCG45L5LRTT3A2CN6BOVE5OI7SEA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:44.934086051 (1h58m0.715019421s) [noise=true , address=45.91.116.8:4048 , piece_id=PCNJHK3FGP2PO2DSHSTIHJ6PGR4ERWS6XW3WTWWIURBYGX4RYF3A, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:44.934364421 (1h58m0.715297801s) [noise=true , address=77.90.31.21:4879 , piece_id=VVX5RS3HRCFW46VDPA6EWS5XPX4VNRPNEO4OA6ZOOXIACC5Y7GUQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:44.934597398 (1h58m0.715530763s) [noise=true , address=45.91.116.7:4010 , piece_id=FPSXDJGHNIJF6RHZ4TPIW3B4O6VT3VHE2RFECVTJCTBJVT57Y52A, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:44.934876966 (1h58m0.715810333s) [noise=true , address=77.90.31.14:4661 , piece_id=GJGHG6U6KNCISVHE6R74LZV6FKUO65T4O6ENJSFOEIDQ6QU2VLAA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:44.935305336 (1h58m0.716238703s) [noise=true , address=77.90.31.14:4662 , piece_id=XYGW55XYY3Q5UP3TWQAA6GFZDVSGKOQOK44RRLZNBM2ODSMXKK4Q, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:44.935406366 (1h58m0.716339729s) [noise=true , address=45.91.116.13:4155 , piece_id=UHF77UOL55YIPUQROVVXOOEMRBKMTMOCOGQZSHDUYXUKSUVQBFSQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:44.936035430 (1h58m0.716968796s) [noise=true , address=77.90.31.15:4698 , piece_id=BJ36WSTZOGQWMIDB6NNGPSS66J5QA7VEQM5ZIVGCKRPRR6OXYT2Q, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:44.937906062 (1h58m0.71883943s) [noise=true , address=77.90.31.11:4553 , piece_id=P65FDAFZYLPPATJ5JNP7KQ4MVOR7K4HIVZVDZPRI4I32NFIETDWQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:45.005296059 (1h58m0.786229488s) [noise=true , address=45.91.116.9:4262 , piece_id=K5NV5HLMNZFHB7PEOE2STOFG44GB3DS7V56BTOIHFXPY2MYREWBA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:49:46.348890347 (1h58m2.129823708s) [noise=true , address=77.90.31.12:4592 , piece_id=UFA3FAYUMNPK6W54LHM3S5CL5L7DD5GOABZ6XRGEY4WLQMSJIELQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil> Jul 26 23:50:14.749321210 (1h58m30.530254587s) [noise=true , address=104.36.82.6:4875 , piece_id=4GPNO2DZZ2HLNE4DGZVT2AKYTYP5ODGF7VIXAXPFBN7OGTVKM5TQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:14.749380296 (1h58m30.530313663s) [noise=true , address=104.36.82.15:4259 , piece_id=IP2ME27ARN24X6SQIWJ4VJZO3DRT3QR5X62D3PXAGPKJSZLJJBJQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:14.749459499 (1h58m30.530392869s) [noise=true , address=104.36.82.15:4257 , piece_id=FYEG6JRAG6GX4OC3TJO553FLTXVX4JUHJU4VAFZHYCCOKOGA5QLA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:14.749475904 (1h58m30.530409282s) [noise=true , address=104.36.82.5:4839 , piece_id=FOFV7KSLZQRGIYF34ROV7QWSN2NZTUVHP3CKKGTBIPKRIQ3T3PCA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:43694->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:14.749526216 (1h58m30.530459584s) [noise=true , address=104.36.82.11:4113 , piece_id=O3OLP35I72UTEBMZUCUDLKPMYCZ4VN7AQNP7W2Z5DPS3UCL4EZLQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:43682->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:14.749563181 (1h58m30.53049656s) [seg_pos={0 0}, upload=1806]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:50:14.774834627 (1h58m30.555768005s) [upload=1806]: Done uploading segments... Jul 26 23:50:14.774880812 (1h58m30.555814191s) [upload=1806]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.028673544s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...............X] duration: 30.002197301s FAIL (2025-07-26 23:50:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002301338s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [X....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029019682s FAIL (2025-07-26 23:49:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 1h57m6.739595004s) [noise=true , address=45.91.116.14:4206 , piece_id=X4OAWUEJY3QS45ZVQAHOCYN5ZQOR5ZMYR443QXRRSLT3H4VLI4AQ, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.958693285 (1h57m6.739626669s) [noise=true , address=77.90.31.19:4847 , piece_id=IHRYNTEY5SZRTUFVUFTEWKFR2SVHYLSAD3HYGVU6L2SHWSJUIHAA, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.959149382 (1h57m6.740082765s) [noise=true , address=64.49.13.139:4478 , piece_id=ISTDHHKCQ6EF7BRSQUHJLGZSBPPRCASFCWWFFZYYTON4MKJTTKZA, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.959265162 (1h57m6.740198551s) [noise=true , address=104.36.82.47:4543 , piece_id=ABZTEPO5KGDEBMQ3UWCSAY3D6JRRQAKQWBZRNLHJZKONULZPSW4A, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.959543419 (1h57m6.740476804s) [noise=true , address=79.127.201.215:4351 , piece_id=YIJLPXVXGF2D5WIGTQDV7EKH6AFESBCY5NYQ3S4P66BOGDZGKUYA, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.959832987 (1h57m6.740766372s) [noise=true , address=77.90.31.11:4576 , piece_id=2DMCVAKN6J2YQAOVEZR4S3CN7PF44MSOOG7BYMUWKV3XFQDFI7PQ, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.959904226 (1h57m6.740837611s) [noise=true , address=104.36.82.52:4090 , piece_id=WCOJD7FO6YJHAP4R637EXSQU44BLHTDOUO5EAFMTVPQI7GZG7RRQ, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.960384541 (1h57m6.741317922s) [noise=true , address=104.36.82.54:4160 , piece_id=ODTJMEQZWBCGFHQNQ72XGQXN2M3NL3N2B2LRTHD7XOUVIMY7BIEQ, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.960451153 (1h57m6.74138454s) [noise=true , address=104.36.82.56:4256 , piece_id=JJBUSYWPGBCIE3OBVMXBESRF626KV4NJCI2GTNEAPXJPTVIWGJLQ, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.960758939 (1h57m6.741692324s) [noise=true , address=77.90.31.17:4792 , piece_id=BMPWWVSWPTNBVAAMSIW634HRJMK5TKHTXCV4OF6H4RAJ7PKPVNWA, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.960957186 (1h57m6.741890574s) [noise=true , address=64.49.13.140:4508 , piece_id=6R2AGSOK7DH3JW3VHLICW6AMHHMEN2GZ6XA7WH5NAOZ2AC4TKEPQ, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.961128404 (1h57m6.742061786s) [noise=true , address=77.90.31.10:4533 , piece_id=2JYCWGX5GS2AXPH34WXSMGXBVLFHWL44P2XTLOXIXSSLDV7CGMYA, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.961427257 (1h57m6.742360641s) [noise=true , address=79.127.219.51:4210 , piece_id=CC5CHBXUKQLGSSPBGIXA65PGMMWHFV2SPRVBA2MUU2O5KMISQ2BA, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.961442268 (1h57m6.742375645s) [noise=true , address=104.36.82.46:4502 , piece_id=W3UD5L4UKN5QN455GL6FWVXRQR24P4DOSQ3IV6HDYUHAIDPNLWDQ, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.961932144 (1h57m6.74286552s) [noise=true , address=64.49.13.154:4927 , piece_id=AFYLWX5GM2YKFBFCVH5CGDBXNHAC63FGDG6HCGPMWO34DFNQC54Q, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.962412839 (1h57m6.743346239s) [noise=true , address=45.91.116.9:4277 , piece_id=WQGSIP6V3AD3LJGC6EGS7GCWVXNDXV5MXLBQ35GBQVTBHTMILU7A, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.962831440 (1h57m6.743764832s) [noise=true , address=104.36.82.53:4142 , piece_id=MVNJG7XUS56IPKO3QSJU7MULKLFO2JAFZHQJTSVSMJVA3VWEZY3A, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.964832151 (1h57m6.74576552s) [noise=true , address=64.49.13.137:4502 , piece_id=GAZZKUWSRW6MNOCVGDP7IHI2SJW3KN5TH4R26FRRQZ2CHFXMXJXQ, seg_pos={0 0}, upload=1797]: Uploading piece... Jul 26 23:48:50.992825199 (1h57m6.773758575s) [noise=true , address=77.90.31.19:4847 , piece_id=IHRYNTEY5SZRTUFVUFTEWKFR2SVHYLSAD3HYGVU6L2SHWSJUIHAA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: storage node overloaded, request limit: 2000 Jul 26 23:48:51.026699454 (1h57m6.807632823s) [upload=1797]: Next returned nil segment Jul 26 23:48:51.026720993 (1h57m6.807654367s) [upload=1797]: Waiting for error group managing segments... Jul 26 23:48:51.032996258 (1h57m6.813929627s) [noise=true , address=77.90.31.10:4533 , piece_id=2JYCWGX5GS2AXPH34WXSMGXBVLFHWL44P2XTLOXIXSSLDV7CGMYA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:48:51.071786016 (1h57m6.852719395s) [noise=true , address=77.90.31.11:4576 , piece_id=2DMCVAKN6J2YQAOVEZR4S3CN7PF44MSOOG7BYMUWKV3XFQDFI7PQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:48:51.107847016 (1h57m6.888780392s) [noise=true , address=77.90.31.17:4792 , piece_id=BMPWWVSWPTNBVAAMSIW634HRJMK5TKHTXCV4OF6H4RAJ7PKPVNWA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:48:51.663534169 (1h57m7.444467553s) [noise=true , address=64.49.13.137:4502 , piece_id=GAZZKUWSRW6MNOCVGDP7IHI2SJW3KN5TH4R26FRRQZ2CHFXMXJXQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:48:52.608569918 (1h57m8.389503289s) [noise=true , address=79.127.219.51:4210 , piece_id=CC5CHBXUKQLGSSPBGIXA65PGMMWHFV2SPRVBA2MUU2O5KMISQ2BA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:48:52.659046744 (1h57m8.439980119s) [noise=true , address=64.49.13.154:4927 , piece_id=AFYLWX5GM2YKFBFCVH5CGDBXNHAC63FGDG6HCGPMWO34DFNQC54Q, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k, address:64.49.13.154:4927): protocol: storage node overloaded, request limit: 2000 Jul 26 23:48:53.697277228 (1h57m9.478210607s) [noise=true , address=64.49.13.140:4508 , piece_id=6R2AGSOK7DH3JW3VHLICW6AMHHMEN2GZ6XA7WH5NAOZ2AC4TKEPQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:48:54.224036958 (1h57m10.004970335s) [noise=true , address=45.91.116.14:4206 , piece_id=X4OAWUEJY3QS45ZVQAHOCYN5ZQOR5ZMYR443QXRRSLT3H4VLI4AQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:48:55.495228481 (1h57m11.276161862s) [noise=true , address=64.49.13.139:4478 , piece_id=ISTDHHKCQ6EF7BRSQUHJLGZSBPPRCASFCWWFFZYYTON4MKJTTKZA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:49:06.295320598 (1h57m22.076253981s) [noise=true , address=45.91.116.9:4277 , piece_id=WQGSIP6V3AD3LJGC6EGS7GCWVXNDXV5MXLBQ35GBQVTBHTMILU7A, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:49:20.248961587 (1h57m36.029894955s) [noise=true , address=79.127.201.215:4351 , piece_id=YIJLPXVXGF2D5WIGTQDV7EKH6AFESBCY5NYQ3S4P66BOGDZGKUYA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil> Jul 26 23:49:50.886255050 (1h58m6.66718844s) [noise=true , address=104.36.82.53:4142 , piece_id=MVNJG7XUS56IPKO3QSJU7MULKLFO2JAFZHQJTSVSMJVA3VWEZY3A, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: rpc: dial tcp 104.36.82.53:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:50.886337272 (1h58m6.667270648s) [noise=true , address=104.36.82.52:4090 , piece_id=WCOJD7FO6YJHAP4R637EXSQU44BLHTDOUO5EAFMTVPQI7GZG7RRQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 159.223.143.39:59390->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:50.886412042 (1h58m6.667345411s) [noise=true , address=104.36.82.54:4160 , piece_id=ODTJMEQZWBCGFHQNQ72XGQXN2M3NL3N2B2LRTHD7XOUVIMY7BIEQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:104.36.82.54:4160): protocol: rpc: dial tcp 104.36.82.54:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:50.886447846 (1h58m6.667381211s) [noise=true , address=104.36.82.56:4256 , piece_id=JJBUSYWPGBCIE3OBVMXBESRF626KV4NJCI2GTNEAPXJPTVIWGJLQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:50.886491267 (1h58m6.667424633s) [noise=true , address=104.36.82.47:4543 , piece_id=ABZTEPO5KGDEBMQ3UWCSAY3D6JRRQAKQWBZRNLHJZKONULZPSW4A, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: write tcp 159.223.143.39:45114->104.36.82.47:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:50.886511678 (1h58m6.667445044s) [noise=true , address=104.36.82.46:4502 , piece_id=W3UD5L4UKN5QN455GL6FWVXRQR24P4DOSQ3IV6HDYUHAIDPNLWDQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: write tcp 159.223.143.39:49514->104.36.82.46:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:50.886566922 (1h58m6.667500299s) [seg_pos={0 0}, upload=1797]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:49:50.912933369 (1h58m6.693866744s) [upload=1797]: Done uploading segments... Jul 26 23:49:50.912980684 (1h58m6.693914066s) [upload=1797]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.029042668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X...X..XX.X...X..X.......X...X....XX..............X.X......] duration: 29.105241524s FAIL (2025-07-26 23:49:29 UTC) context deadline exceeded Duration: 29.105268146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.032334886s FAIL (2025-07-26 23:49:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.032367731s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 30.026370872s FAIL (2025-07-26 23:48:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== iled (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:48126->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498548008 (1h56m23.279481386s) [seg_pos={0 0}, upload=1777]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:48:07.526079352 (1h56m23.307012724s) [upload=1777]: Done uploading segments... Jul 26 23:48:07.526127435 (1h56m23.30706082s) [upload=1777]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:48:29.973052621 (1h56m45.753985989s) [upload=1791]: Starting upload Jul 26 23:48:29.973154855 (1h56m45.754088241s) [upload=1791]: Done starting upload Jul 26 23:48:29.973294734 (1h56m45.7542281s) [upload=1791]: Uploading segments... Jul 26 23:48:29.973657027 (1h56m45.754590393s) [upload=1791]: Got next segment. Inline: false Jul 26 23:48:29.973721230 (1h56m45.754654598s) [upload=1791]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:48:29.973728168 (1h56m45.75466153s) [upload=1791]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:48:30.001024890 (1h56m45.781958302s) [seg_pos={0 0}, upload=1791]: Begin upload segment... Jul 26 23:48:30.001084651 (1h56m45.78201805s) [seg_pos={0 0}, upload=1791]: Done begin upload segment. Jul 26 23:48:30.001117532 (1h56m45.782050935s) [noise=true , address=45.91.116.11:4117 , piece_id=D34BCNYYKIPKEOVK7NVX2FLZJL7HMK6BZNAIRE2MIG4OZ2F4J7MA, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.001651152 (1h56m45.782584553s) [noise=true , address=45.91.116.13:4153 , piece_id=GTVBZEX6OHDVD6HYQR3G6TR55V4YOQIGGMSTLSB56S33HJ2ETV5A, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.002016296 (1h56m45.78294969s) [noise=true , address=45.91.116.6:4973 , piece_id=SPO3XWPX5FUJFFSOEW5ARV2INMOS24APEZZWNYAOQDV6IH6MD2CQ, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.002367862 (1h56m45.783301264s) [noise=true , address=104.36.82.14:4221 , piece_id=HBYBZTSMBVVJWDDWOMFQCMIWAQJGSG2YLGHL22NVRR5BBG4WAB2A, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.002688708 (1h56m45.783622086s) [noise=true , address=104.36.82.8:4971 , piece_id=L3BE65HJ3WTOT7ZIYD5UUSWO5C5JS4UX7LBYA45KJBLJYRHML3ZQ, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.002989049 (1h56m45.783922431s) [noise=true , address=45.91.116.12:4227 , piece_id=C266SVGBNE4QKDPFAMPWNAJX4GOLGNRQWUP2SUX6CCMSNI3AVEIQ, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.003351875 (1h56m45.784285256s) [noise=true , address=45.91.116.10:4081 , piece_id=OTTOIN5DEVRQYWMJ4VF54US7IVBECSJZU5OSV22XKRVXMXZP6DRA, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.005317512 (1h56m45.786250893s) [noise=true , address=104.36.82.6:4874 , piece_id=GQDCX4I2PWKO76IAUPFMKZ6P34CUCHAFKP6VG2DZTV2H33M2KATQ, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.005821370 (1h56m45.78675475s) [noise=true , address=45.91.116.8:4045 , piece_id=V33YGB2H5OGN2O4IE4M5NJV4F5AUVTFHGOUD2BHUMEOBFX6VYVNA, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.006147011 (1h56m45.78708039s) [noise=true , address=104.36.82.9:4006 , piece_id=AEBIZNESPA42354VZIGS3EGUZKDKZ5EAYBBTKDHRJF7A6RHHUL4A, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.006457281 (1h56m45.787390664s) [noise=true , address=45.91.116.8:4048 , piece_id=SJ5LEGP3XQJ32O5HVB5XDYJHG32BWNVUZFTW3GL5UBBHFOW57UQA, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.006754649 (1h56m45.787688032s) [noise=true , address=45.91.116.5:4298 , piece_id=4OFZU66WCOPNSULSKZUZODLS4AIWYBYRBVQVL35SP3L6FCNBN2AA, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.007096765 (1h56m45.788030148s) [noise=true , address=77.90.31.14:4663 , piece_id=TQZDFBBCYQUIBKJTFCQ5TJXCGRPAZJRFGU5SELPWN2XTE4Y3BMTA, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.007491917 (1h56m45.788425299s) [noise=true , address=45.91.116.7:4010 , piece_id=5UL3B7UEM6EPYHU7AGMWSHU6TAGRDBNEU25MX4OUC4LZPP4OPBMA, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.009584814 (1h56m45.790518222s) [noise=true , address=104.36.82.3:4765 , piece_id=UGUHNPRLC56BYOYDFI466SYFRCFEDBUD4ICO2DVKI3CPO7QAQNUQ, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.010187976 (1h56m45.791121383s) [noise=true , address=104.36.82.13:4187 , piece_id=R6CHN46X2IXV3KMYOED2O7PP34TSB45ZWPBJHWNZT4PYHF7LECZQ, seg_pos={0 0}, upload=1791]: Uploading piece... Jul 26 23:48:30.274819250 (1h56m46.055752635s) [upload=1791]: Next returned nil segment Jul 26 23:48:30.274844994 (1h56m46.055778383s) [upload=1791]: Waiting for error group managing segments... Jul 26 23:48:30.282908830 (1h56m46.063842198s) [noise=true , address=45.91.116.5:4298 , piece_id=4OFZU66WCOPNSULSKZUZODLS4AIWYBYRBVQVL35SP3L6FCNBN2AA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.284035898 (1h56m46.064969266s) [noise=true , address=45.91.116.10:4081 , piece_id=OTTOIN5DEVRQYWMJ4VF54US7IVBECSJZU5OSV22XKRVXMXZP6DRA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.285029686 (1h56m46.065963049s) [noise=true , address=45.91.116.8:4045 , piece_id=V33YGB2H5OGN2O4IE4M5NJV4F5AUVTFHGOUD2BHUMEOBFX6VYVNA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.285997064 (1h56m46.066930428s) [noise=true , address=45.91.116.12:4227 , piece_id=C266SVGBNE4QKDPFAMPWNAJX4GOLGNRQWUP2SUX6CCMSNI3AVEIQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.286768904 (1h56m46.067702273s) [noise=true , address=45.91.116.8:4048 , piece_id=SJ5LEGP3XQJ32O5HVB5XDYJHG32BWNVUZFTW3GL5UBBHFOW57UQA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.287375432 (1h56m46.068308797s) [noise=true , address=77.90.31.14:4663 , piece_id=TQZDFBBCYQUIBKJTFCQ5TJXCGRPAZJRFGU5SELPWN2XTE4Y3BMTA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.287978896 (1h56m46.068912263s) [noise=true , address=45.91.116.6:4973 , piece_id=SPO3XWPX5FUJFFSOEW5ARV2INMOS24APEZZWNYAOQDV6IH6MD2CQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.288573535 (1h56m46.069506915s) [noise=true , address=45.91.116.13:4153 , piece_id=GTVBZEX6OHDVD6HYQR3G6TR55V4YOQIGGMSTLSB56S33HJ2ETV5A, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.292985744 (1h56m46.073919125s) [noise=true , address=45.91.116.7:4010 , piece_id=5UL3B7UEM6EPYHU7AGMWSHU6TAGRDBNEU25MX4OUC4LZPP4OPBMA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:30.297040331 (1h56m46.077973697s) [noise=true , address=45.91.116.11:4117 , piece_id=D34BCNYYKIPKEOVK7NVX2FLZJL7HMK6BZNAIRE2MIG4OZ2F4J7MA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil> Jul 26 23:48:59.937478845 (1h57m15.718412224s) [noise=true , address=104.36.82.14:4221 , piece_id=HBYBZTSMBVVJWDDWOMFQCMIWAQJGSG2YLGHL22NVRR5BBG4WAB2A, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:59.937695728 (1h57m15.718629107s) [noise=true , address=104.36.82.3:4765 , piece_id=UGUHNPRLC56BYOYDFI466SYFRCFEDBUD4ICO2DVKI3CPO7QAQNUQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:59.937763042 (1h57m15.718696419s) [noise=true , address=104.36.82.9:4006 , piece_id=AEBIZNESPA42354VZIGS3EGUZKDKZ5EAYBBTKDHRJF7A6RHHUL4A, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:48:59.937839038 (1h57m15.718772415s) [noise=true , address=104.36.82.13:4187 , piece_id=R6CHN46X2IXV3KMYOED2O7PP34TSB45ZWPBJHWNZT4PYHF7LECZQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:38866->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:59.937855239 (1h57m15.718788619s) [noise=true , address=104.36.82.8:4971 , piece_id=L3BE65HJ3WTOT7ZIYD5UUSWO5C5JS4UX7LBYA45KJBLJYRHML3ZQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:59.937876625 (1h57m15.718809995s) [noise=true , address=104.36.82.6:4874 , piece_id=GQDCX4I2PWKO76IAUPFMKZ6P34CUCHAFKP6VG2DZTV2H33M2KATQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:59.937906032 (1h57m15.718839395s) [seg_pos={0 0}, upload=1791]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:48:59.961543129 (1h57m15.742476505s) [upload=1791]: Done uploading segments... Jul 26 23:48:59.961589585 (1h57m15.742522967s) [upload=1791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.02639934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.031329045s FAIL (2025-07-26 23:48:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.031352418s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026877216s FAIL (2025-07-26 23:48:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ts... Jul 26 23:47:50.818338951 (1h56m6.599272339s) [upload=1781]: Got next segment. Inline: false Jul 26 23:47:50.818427169 (1h56m6.599360546s) [upload=1781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:47:50.818435467 (1h56m6.599368852s) [upload=1781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:47:50.852575146 (1h56m6.633508512s) [seg_pos={0 0}, upload=1781]: Begin upload segment... Jul 26 23:47:50.852632775 (1h56m6.633566153s) [seg_pos={0 0}, upload=1781]: Done begin upload segment. Jul 26 23:47:50.852672162 (1h56m6.63360554s) [noise=true , address=77.90.31.16:4747 , piece_id=HRMAPCAJ4Q7JQPTALZQCHSKI43P6Q54D45IZJQXT6YP5U5FFOKRA, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.853143012 (1h56m6.634076403s) [noise=true , address=104.36.82.55:4198 , piece_id=SXGZYS3YKOZC5GJOOOI6BBWUVIKM4NE25C3GVUXNLEP5STQPIC5A, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.853673148 (1h56m6.634606532s) [noise=true , address=104.36.82.53:4138 , piece_id=RDOXJNWARUJ2Q5FWGQ7XNMPYSAQMRYWTECPSRV2DB4IKHNPEWSKQ, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.853845836 (1h56m6.634779214s) [noise=true , address=77.90.31.13:4644 , piece_id=QJLD7E6JMGETVW46TNJJ2ZJJNTESFUTWVOZ4V24VYBMYCPGVPE3Q, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.854729082 (1h56m6.635662462s) [noise=true , address=79.127.201.214:4289 , piece_id=5Q272LELGSMG73ITNPPWSU6KRZGJVUXSNJGJL3N2VJLH6E2XD47Q, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.855582594 (1h56m6.63651598s) [noise=true , address=79.127.205.245:4243 , piece_id=MGXEDD7MBPK6WHOZE272GP3AQUJNGD2PK22VTTVVB5GGZBMWZGNA, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.856754050 (1h56m6.637687434s) [noise=true , address=79.127.201.215:4359 , piece_id=W5TFF2OVPLMU5L5XIV6JSPLKL3HT7YM4EUX4M55VFUFIOKM7CFPQ, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.857246574 (1h56m6.638179959s) [noise=true , address=79.127.220.101:4048 , piece_id=5Y7GEA54Q5S7Y63J5Z5IUREPQIFPZGLHX3ZSXL3DJUCVXWQE2QTA, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.858576749 (1h56m6.639510153s) [noise=true , address=104.36.82.56:4233 , piece_id=MGTZAU7DY4UU3RPIF7PJ4IT6FOOH6YNQ4ULYC5HGGDRKIIDHZVNA, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.859163927 (1h56m6.640097308s) [noise=true , address=104.36.82.46:4507 , piece_id=FFRVQS72UBWHSNCTQWXCQU6VMRUGDA7FJPVUVPZW4G3MVW6WD3ZA, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.859536418 (1h56m6.640469796s) [noise=true , address=104.36.82.47:4544 , piece_id=42MEVXRAN3KXUJJWRWTDC3W2IY27B6CSKZSYMHWA6MDW6GATI67A, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.859853040 (1h56m6.640786418s) [noise=true , address=77.90.31.10:4533 , piece_id=TJKDZQQGYYGBMJVJDXUZMB2XRLVFEO6YXDZG6UEZWASL2OQCCLCQ, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.860218627 (1h56m6.641151992s) [noise=true , address=79.127.220.101:4041 , piece_id=PZ2QP3GERIHRGI2BGDNLASEKHKUJJ7NEYYD73IDKEH44PH6WWP4Q, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.860692805 (1h56m6.641626192s) [noise=true , address=64.49.13.154:4927 , piece_id=KXGJSF6PWXRFAPFYUDDO75ZHQEP524RFMJFZ6UYDVPJIAQMYYQBQ, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.861174941 (1h56m6.642108311s) [noise=true , address=79.127.205.235:4666 , piece_id=4LH3KM55VESEPSIXU6VGEBI4ZGMSPUVJ7KZXTFM4PNJEZCW2Q6RQ, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.861945352 (1h56m6.642878738s) [noise=true , address=79.127.219.38:4017 , piece_id=7USCBFN3SR5U434ES632SUKSEZFB44CXIXULUXP447CEROVTK4GA, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.864719645 (1h56m6.645653014s) [noise=true , address=45.91.116.5:4306 , piece_id=JTJQ3RMG3KQDDRKRWKQ2HEZOWKCWEKE2TBBLYZPLOQBWBCW5LEPQ, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.865188257 (1h56m6.646121617s) [noise=true , address=45.91.116.9:4267 , piece_id=NAGNQNRQVMOYPUOP5RJ7OFKTII6IV6KRB3KSYTRKIS3VTOZMWA5A, seg_pos={0 0}, upload=1781]: Uploading piece... Jul 26 23:47:50.967850852 (1h56m6.74878424s) [upload=1781]: Next returned nil segment Jul 26 23:47:50.967909997 (1h56m6.748843386s) [upload=1781]: Waiting for error group managing segments... Jul 26 23:47:50.973153501 (1h56m6.754086867s) [noise=true , address=77.90.31.10:4533 , piece_id=TJKDZQQGYYGBMJVJDXUZMB2XRLVFEO6YXDZG6UEZWASL2OQCCLCQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:50.973823257 (1h56m6.754756627s) [noise=true , address=79.127.219.38:4017 , piece_id=7USCBFN3SR5U434ES632SUKSEZFB44CXIXULUXP447CEROVTK4GA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:51.229992473 (1h56m7.010925843s) [noise=true , address=45.91.116.9:4267 , piece_id=NAGNQNRQVMOYPUOP5RJ7OFKTII6IV6KRB3KSYTRKIS3VTOZMWA5A, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:51.522582029 (1h56m7.303515396s) [noise=true , address=64.49.13.154:4927 , piece_id=KXGJSF6PWXRFAPFYUDDO75ZHQEP524RFMJFZ6UYDVPJIAQMYYQBQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:51.528512108 (1h56m7.309445491s) [noise=true , address=79.127.205.245:4243 , piece_id=MGXEDD7MBPK6WHOZE272GP3AQUJNGD2PK22VTTVVB5GGZBMWZGNA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:51.663423094 (1h56m7.444356473s) [noise=true , address=79.127.205.235:4666 , piece_id=4LH3KM55VESEPSIXU6VGEBI4ZGMSPUVJ7KZXTFM4PNJEZCW2Q6RQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:54.327215338 (1h56m10.108148699s) [noise=true , address=77.90.31.13:4644 , piece_id=QJLD7E6JMGETVW46TNJJ2ZJJNTESFUTWVOZ4V24VYBMYCPGVPE3Q, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:54.694537518 (1h56m10.475470891s) [noise=true , address=79.127.220.101:4048 , piece_id=5Y7GEA54Q5S7Y63J5Z5IUREPQIFPZGLHX3ZSXL3DJUCVXWQE2QTA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:56.141286870 (1h56m11.922220239s) [noise=true , address=45.91.116.5:4306 , piece_id=JTJQ3RMG3KQDDRKRWKQ2HEZOWKCWEKE2TBBLYZPLOQBWBCW5LEPQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:56.627782361 (1h56m12.408715827s) [noise=true , address=79.127.220.101:4041 , piece_id=PZ2QP3GERIHRGI2BGDNLASEKHKUJJ7NEYYD73IDKEH44PH6WWP4Q, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:47:57.625636065 (1h56m13.406569431s) [noise=true , address=79.127.201.214:4289 , piece_id=5Q272LELGSMG73ITNPPWSU6KRZGJVUXSNJGJL3N2VJLH6E2XD47Q, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:48:32.739574388 (1h56m48.520507752s) [noise=true , address=79.127.201.215:4359 , piece_id=W5TFF2OVPLMU5L5XIV6JSPLKL3HT7YM4EUX4M55VFUFIOKM7CFPQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil> Jul 26 23:48:50.787580192 (1h57m6.568513574s) [noise=true , address=104.36.82.47:4544 , piece_id=42MEVXRAN3KXUJJWRWTDC3W2IY27B6CSKZSYMHWA6MDW6GATI67A, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:104.36.82.47:4544): protocol: rpc: dial tcp 104.36.82.47:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:50.787657420 (1h57m6.568590797s) [noise=true , address=104.36.82.55:4198 , piece_id=SXGZYS3YKOZC5GJOOOI6BBWUVIKM4NE25C3GVUXNLEP5STQPIC5A, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: write tcp 159.223.143.39:47734->104.36.82.55:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:50.787706646 (1h57m6.568640029s) [noise=true , address=77.90.31.16:4747 , piece_id=HRMAPCAJ4Q7JQPTALZQCHSKI43P6Q54D45IZJQXT6YP5U5FFOKRA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: rpc: dial tcp 77.90.31.16:4747: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:50.787812939 (1h57m6.568746314s) [noise=true , address=104.36.82.53:4138 , piece_id=RDOXJNWARUJ2Q5FWGQ7XNMPYSAQMRYWTECPSRV2DB4IKHNPEWSKQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:50.787891428 (1h57m6.568824806s) [noise=true , address=104.36.82.56:4233 , piece_id=MGTZAU7DY4UU3RPIF7PJ4IT6FOOH6YNQ4ULYC5HGGDRKIIDHZVNA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:50.787922971 (1h57m6.568856347s) [noise=true , address=104.36.82.46:4507 , piece_id=FFRVQS72UBWHSNCTQWXCQU6VMRUGDA7FJPVUVPZW4G3MVW6WD3ZA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:50.787962855 (1h57m6.568896242s) [seg_pos={0 0}, upload=1781]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:48:50.812333363 (1h57m6.593266727s) [upload=1781]: Done uploading segments... Jul 26 23:48:50.812374174 (1h57m6.593307549s) [upload=1781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.026896411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...............] duration: 1.931986371s FAIL (2025-07-26 23:48:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.932012676s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..X...X..XX.X...X..X.......X...X....XX..............X.X.....] duration: 35.447672495s FAIL (2025-07-26 23:48:43 UTC) context deadline exceeded Duration: 35.447714062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XXX.X....XX.X.X.X.......XXX...XX.....X.XXX..............] duration: 526.756777ms FAIL (2025-07-26 23:48:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 526.782488ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 30.030261844s FAIL (2025-07-26 23:48:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 6]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:47:37.527875473 (1h55m53.308808849s) [upload=1777]: Starting upload Jul 26 23:47:37.527949599 (1h55m53.308882979s) [upload=1777]: Done starting upload Jul 26 23:47:37.528017058 (1h55m53.308950424s) [upload=1777]: Uploading segments... Jul 26 23:47:37.528294490 (1h55m53.309227861s) [upload=1777]: Got next segment. Inline: false Jul 26 23:47:37.528338994 (1h55m53.309272369s) [upload=1777]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:47:37.528344859 (1h55m53.309278235s) [upload=1777]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:47:37.558168809 (1h55m53.339102191s) [seg_pos={0 0}, upload=1777]: Begin upload segment... Jul 26 23:47:37.558235107 (1h55m53.339168486s) [seg_pos={0 0}, upload=1777]: Done begin upload segment. Jul 26 23:47:37.558287934 (1h55m53.339221325s) [noise=true , address=45.91.116.5:4297 , piece_id=LF4VTRQS5KUBY5ZMDJIVCGS3BCMRV7ASOGJU33MFDEP4KIPPTXNQ, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.558942976 (1h55m53.339876363s) [noise=true , address=45.91.116.8:4047 , piece_id=O4IQUREEYILXI5DBT7NQWBPTBYLT4J4OGAOYDQ4AENELVIOYIEGA, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.559380729 (1h55m53.340314113s) [noise=true , address=104.36.82.5:4839 , piece_id=44Y5WSP5THZCBZEAAN2UAOEV6SWVKGJKNUWNB2XK4CBNRYOK6CDQ, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.559759875 (1h55m53.34069326s) [noise=true , address=77.90.31.17:4770 , piece_id=ASY7Z3FGRPLF3LZJSAFHFXGRGXSO2H6QFZ6MJ73IAXVWVYZ3X3MQ, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.560303225 (1h55m53.341236612s) [noise=true , address=104.36.82.11:4115 , piece_id=FFICM37JDSWEMPTX472OOBAJDJBL7DW6RSFRN3T2NAIT6ABFPV5Q, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.560683951 (1h55m53.341617338s) [noise=true , address=77.90.31.18:4807 , piece_id=TWZH6ZGFIHDNPB2VJCWFBUBXNS3E45FRJOO3HW2JIRUWNN5PTSMA, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.561045028 (1h55m53.341978413s) [noise=true , address=45.91.116.8:4048 , piece_id=A62MAJI7XDAJ7FV5Q6T6Z3QEYYAWRW4XLG5AFPL4GTSGNTJMIA2Q, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.561380481 (1h55m53.342313866s) [noise=true , address=104.36.82.8:4969 , piece_id=HKEPNFLA3FGNCJUL5INGDFRKGGZJHEF5YD7SULHHJP5SYIV3J6RQ, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.561750873 (1h55m53.34268426s) [noise=true , address=77.90.31.13:4627 , piece_id=MB6BZ2N7JEQSNT34DZKMQH7QW37SOXMCNKZATWK7WFOZMP4RFOXA, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.562132507 (1h55m53.343065893s) [noise=true , address=104.36.82.5:4837 , piece_id=4N6VJS5MXC2ZEX6K2OAGATLVGXGXAFJF3YGITIE6FMBJ4FRYI7LA, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.562492426 (1h55m53.343425809s) [noise=true , address=104.36.82.2:4732 , piece_id=SO72M3MPNP3P5LOHBSNDFLKH7JGA2FIU5FRAJ5GLQPVHYBOXVKYA, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.562816523 (1h55m53.343749907s) [noise=true , address=104.36.82.15:4257 , piece_id=YIBYEQOCFB2RQBIH25O76NKKCVWAO5YAY6Q6PG3KQPLKD4BIBGYQ, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.563122290 (1h55m53.344055671s) [noise=true , address=104.36.82.10:4079 , piece_id=WVQDLMLEMPQQLYV6ESENXILSBPZ4L2KSCESFOSCMPC5S4ATMSXRA, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.563350758 (1h55m53.344284144s) [noise=true , address=45.91.116.9:4261 , piece_id=TL3FY4IT6PSJLEJFNMFU5P7IU5SPYTY3ZZNYGXGXG4LBTNUNVC5A, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.563844153 (1h55m53.3447776s) [noise=true , address=45.91.116.6:4975 , piece_id=INLDPSR72KZI6UCE7XBBBM7MF3QVUYZK5QENI2MPQZUZX2Y7IW6A, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.564376257 (1h55m53.345309632s) [noise=true , address=45.91.116.12:4227 , piece_id=C7U3VM7JTAPXZL7DGSNAICDFSCVRHSQ3HALSIUDCGTEYAXMRVKFQ, seg_pos={0 0}, upload=1777]: Uploading piece... Jul 26 23:47:37.641178717 (1h55m53.422112098s) [upload=1777]: Next returned nil segment Jul 26 23:47:37.641199954 (1h55m53.42213334s) [upload=1777]: Waiting for error group managing segments... Jul 26 23:47:37.645131321 (1h55m53.426064701s) [noise=true , address=45.91.116.8:4048 , piece_id=A62MAJI7XDAJ7FV5Q6T6Z3QEYYAWRW4XLG5AFPL4GTSGNTJMIA2Q, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil> Jul 26 23:47:37.645740259 (1h55m53.426673637s) [noise=true , address=45.91.116.8:4047 , piece_id=O4IQUREEYILXI5DBT7NQWBPTBYLT4J4OGAOYDQ4AENELVIOYIEGA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil> Jul 26 23:47:37.646303572 (1h55m53.427236951s) [noise=true , address=45.91.116.12:4227 , piece_id=C7U3VM7JTAPXZL7DGSNAICDFSCVRHSQ3HALSIUDCGTEYAXMRVKFQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil> Jul 26 23:47:37.646915711 (1h55m53.427849087s) [noise=true , address=77.90.31.17:4770 , piece_id=ASY7Z3FGRPLF3LZJSAFHFXGRGXSO2H6QFZ6MJ73IAXVWVYZ3X3MQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil> Jul 26 23:47:37.647563748 (1h55m53.428497125s) [noise=true , address=45.91.116.6:4975 , piece_id=INLDPSR72KZI6UCE7XBBBM7MF3QVUYZK5QENI2MPQZUZX2Y7IW6A, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil> Jul 26 23:47:37.985755377 (1h55m53.766688752s) [noise=true , address=45.91.116.9:4261 , piece_id=TL3FY4IT6PSJLEJFNMFU5P7IU5SPYTY3ZZNYGXGXG4LBTNUNVC5A, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil> Jul 26 23:47:38.091323807 (1h55m53.872257177s) [noise=true , address=45.91.116.5:4297 , piece_id=LF4VTRQS5KUBY5ZMDJIVCGS3BCMRV7ASOGJU33MFDEP4KIPPTXNQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil> Jul 26 23:47:39.022083056 (1h55m54.803016433s) [noise=true , address=77.90.31.13:4627 , piece_id=MB6BZ2N7JEQSNT34DZKMQH7QW37SOXMCNKZATWK7WFOZMP4RFOXA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil> Jul 26 23:48:07.498227396 (1h56m23.279160778s) [noise=true , address=104.36.82.5:4839 , piece_id=44Y5WSP5THZCBZEAAN2UAOEV6SWVKGJKNUWNB2XK4CBNRYOK6CDQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498315165 (1h56m23.279248531s) [noise=true , address=104.36.82.2:4732 , piece_id=SO72M3MPNP3P5LOHBSNDFLKH7JGA2FIU5FRAJ5GLQPVHYBOXVKYA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498385439 (1h56m23.279318803s) [noise=true , address=104.36.82.11:4115 , piece_id=FFICM37JDSWEMPTX472OOBAJDJBL7DW6RSFRN3T2NAIT6ABFPV5Q, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498420911 (1h56m23.279354278s) [noise=true , address=104.36.82.10:4079 , piece_id=WVQDLMLEMPQQLYV6ESENXILSBPZ4L2KSCESFOSCMPC5S4ATMSXRA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498457186 (1h56m23.27939055s) [noise=true , address=77.90.31.18:4807 , piece_id=TWZH6ZGFIHDNPB2VJCWFBUBXNS3E45FRJOO3HW2JIRUWNN5PTSMA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498474409 (1h56m23.279407776s) [noise=true , address=104.36.82.8:4969 , piece_id=HKEPNFLA3FGNCJUL5INGDFRKGGZJHEF5YD7SULHHJP5SYIV3J6RQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:42522->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498484500 (1h56m23.279417863s) [noise=true , address=104.36.82.5:4837 , piece_id=4N6VJS5MXC2ZEX6K2OAGATLVGXGXAFJF3YGITIE6FMBJ4FRYI7LA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498521916 (1h56m23.2794553s) [noise=true , address=104.36.82.15:4257 , piece_id=YIBYEQOCFB2RQBIH25O76NKKCVWAO5YAY6Q6PG3KQPLKD4BIBGYQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:48126->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:07.498548008 (1h56m23.279481386s) [seg_pos={0 0}, upload=1777]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:48:07.526079352 (1h56m23.307012724s) [upload=1777]: Done uploading segments... Jul 26 23:48:07.526127435 (1h56m23.30706082s) [upload=1777]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.030287736s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...X...X..XX.X...X..X.......X...X....XX..............X.X....] duration: 34.959670595s FAIL (2025-07-26 23:48:07 UTC) context deadline exceeded Duration: 34.959695117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026385634s FAIL (2025-07-26 23:47:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 23:46:50.741258335 (1h55m6.522191701s) [seg_pos={0 0}, upload=1761]: Done begin upload segment. Jul 26 23:46:50.741296624 (1h55m6.52223s) [noise=true , address=79.127.205.245:4223 , piece_id=XUYPCZKXBDRPPXBJZABJWFUDC4R3HMNWCXKEZB7GRC4AOPMAC53A, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.741863330 (1h55m6.522796711s) [noise=true , address=64.49.13.140:4514 , piece_id=B43VCZPYNZAVEPM73B6ZP4HA7VUTZVH6IWGQPRWZCU6I2XW3VC3A, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.742243093 (1h55m6.523176459s) [noise=true , address=104.36.82.55:4198 , piece_id=MOAJGCTK2HX352AFAV2DPXQB2ABF7TGMLG34EGOBNNZV2SJIR2HA, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.742713710 (1h55m6.523647075s) [noise=true , address=79.127.205.233:4639 , piece_id=IFHAJKNGZI7CXLLHIFZ6UU35OXKN4R6DRKTZOFUTSSWEFLPUD34A, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.742795025 (1h55m6.523728401s) [noise=true , address=79.127.205.235:4691 , piece_id=XD3PQQWMCIF4TCHGG4PCVEHLV46CE4RCBBHDJAM3KZOUFZMNZV5Q, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.743443815 (1h55m6.524377193s) [noise=true , address=64.49.13.138:4482 , piece_id=2T5EVLLGN7S3IKBNBYJPHM6ZOAOFVWJJ73UMN4M7U6M7D5EYRZZA, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.743489781 (1h55m6.524423156s) [noise=true , address=79.127.219.38:4009 , piece_id=MXCAEBV2EASVTXBV2ODL7PB3BTKYBLIBX2P6AMH6QLWBSONLEWPA, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.745542940 (1h55m6.526476328s) [noise=true , address=104.36.82.53:4155 , piece_id=VLRJ3PFP33L65VJBDGNDAFDC2OZLYQ3QXUFLNJZAVUWGRXWHQWLQ, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.747456571 (1h55m6.52838996s) [noise=true , address=77.90.31.16:4751 , piece_id=IKPXMQS4NK5SPH4OG7VVYEZPURDYOX42VNONP3NQKGVQBIASWYJA, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.748561713 (1h55m6.52949509s) [noise=true , address=77.90.31.11:4562 , piece_id=Q6XE2QDQHIN2V3FVP5ND5NJ775IOOFGPGG2KNMXCJOG3UDIS2QPA, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.751253569 (1h55m6.532186934s) [noise=true , address=104.36.82.47:4550 , piece_id=JRR64PPLW3XFFLKH6DH2TEQSJ3CEOKNJN4JNCLK6DN66KCNX6DAA, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.751855586 (1h55m6.532788966s) [noise=true , address=79.127.205.246:4282 , piece_id=T6QL3EUTWM764VCXJ66PE2SQLE26XOWFYGESUQRUVVXMRGJ4A3JA, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.752829775 (1h55m6.533763161s) [noise=true , address=104.36.82.3:4796 , piece_id=2E73WPDAPO7SHJQUH7NVBDAL37AKZCKJ5Z2DPLDSDGHYMSAXS7CQ, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.753702355 (1h55m6.534635741s) [noise=true , address=79.127.205.234:4615 , piece_id=PFH2OVDWWOIHDRMJLNEE5JQPZEB525CLCDHBL7XLNHL65ANJRZKQ, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.755846908 (1h55m6.536780292s) [noise=true , address=104.36.82.14:4236 , piece_id=OGTQF76VCS6ZZ37IFULCSOOY57IY52N6JSW5M7AYIVW7NREZR5VA, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.758149954 (1h55m6.539083344s) [noise=true , address=45.91.116.14:4194 , piece_id=ZN7V6Y6OXZSGI55LRZYGR7CCLMXFAI3QFBT6DFU27EFG7OYVZZYQ, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.762999165 (1h55m6.543932552s) [noise=true , address=104.36.82.56:4235 , piece_id=B5OKANQZ3S6MBNTNJBD75HLRJRFKZF6GVCJSEOTH4P4XTFBQ352A, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.768688898 (1h55m6.549622286s) [noise=true , address=104.36.82.46:4483 , piece_id=6T6VBFPV4WKYFHPFBMIM2QIJZZZ666PVQ7OCDBFGFRNIWXYTUH6A, seg_pos={0 0}, upload=1761]: Uploading piece... Jul 26 23:46:50.830056987 (1h55m6.610990363s) [upload=1761]: Next returned nil segment Jul 26 23:46:50.830077461 (1h55m6.61101084s) [upload=1761]: Waiting for error group managing segments... Jul 26 23:46:50.835286105 (1h55m6.616219474s) [noise=true , address=79.127.219.38:4009 , piece_id=MXCAEBV2EASVTXBV2ODL7PB3BTKYBLIBX2P6AMH6QLWBSONLEWPA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:50.844324478 (1h55m6.625257848s) [noise=true , address=45.91.116.14:4194 , piece_id=ZN7V6Y6OXZSGI55LRZYGR7CCLMXFAI3QFBT6DFU27EFG7OYVZZYQ, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:50.867851587 (1h55m6.64878497s) [noise=true , address=77.90.31.11:4562 , piece_id=Q6XE2QDQHIN2V3FVP5ND5NJ775IOOFGPGG2KNMXCJOG3UDIS2QPA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:51.306073912 (1h55m7.087007281s) [noise=true , address=79.127.205.245:4223 , piece_id=XUYPCZKXBDRPPXBJZABJWFUDC4R3HMNWCXKEZB7GRC4AOPMAC53A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:51.311970486 (1h55m7.092903867s) [noise=true , address=79.127.205.235:4691 , piece_id=XD3PQQWMCIF4TCHGG4PCVEHLV46CE4RCBBHDJAM3KZOUFZMNZV5Q, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:51.320568779 (1h55m7.101502158s) [noise=true , address=79.127.205.233:4639 , piece_id=IFHAJKNGZI7CXLLHIFZ6UU35OXKN4R6DRKTZOFUTSSWEFLPUD34A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:51.338459952 (1h55m7.119393336s) [noise=true , address=79.127.205.246:4282 , piece_id=T6QL3EUTWM764VCXJ66PE2SQLE26XOWFYGESUQRUVVXMRGJ4A3JA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:51.344736585 (1h55m7.125669967s) [noise=true , address=79.127.205.234:4615 , piece_id=PFH2OVDWWOIHDRMJLNEE5JQPZEB525CLCDHBL7XLNHL65ANJRZKQ, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:52.954574066 (1h55m8.735507434s) [noise=true , address=64.49.13.138:4482 , piece_id=2T5EVLLGN7S3IKBNBYJPHM6ZOAOFVWJJ73UMN4M7U6M7D5EYRZZA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:46:53.162653573 (1h55m8.94358695s) [noise=true , address=64.49.13.140:4514 , piece_id=B43VCZPYNZAVEPM73B6ZP4HA7VUTZVH6IWGQPRWZCU6I2XW3VC3A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil> Jul 26 23:47:50.672326331 (1h56m6.453259701s) [noise=true , address=104.36.82.56:4235 , piece_id=B5OKANQZ3S6MBNTNJBD75HLRJRFKZF6GVCJSEOTH4P4XTFBQ352A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: rpc: dial tcp 104.36.82.56:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:50.672426801 (1h56m6.453360169s) [noise=true , address=104.36.82.3:4796 , piece_id=2E73WPDAPO7SHJQUH7NVBDAL37AKZCKJ5Z2DPLDSDGHYMSAXS7CQ, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: rpc: dial tcp 104.36.82.3:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:50.672518748 (1h56m6.453452113s) [noise=true , address=104.36.82.46:4483 , piece_id=6T6VBFPV4WKYFHPFBMIM2QIJZZZ666PVQ7OCDBFGFRNIWXYTUH6A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: rpc: dial tcp 104.36.82.46:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:50.672594467 (1h56m6.453527833s) [noise=true , address=104.36.82.47:4550 , piece_id=JRR64PPLW3XFFLKH6DH2TEQSJ3CEOKNJN4JNCLK6DN66KCNX6DAA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:50.672688561 (1h56m6.453621926s) [noise=true , address=104.36.82.14:4236 , piece_id=OGTQF76VCS6ZZ37IFULCSOOY57IY52N6JSW5M7AYIVW7NREZR5VA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:1XaE29W27ym2RmuGdBvHeKcQZZEUbask1zsm9Bz8xLokZNGfcj, address:104.36.82.14:4236): protocol: rpc: dial tcp 104.36.82.14:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:50.672761864 (1h56m6.453695229s) [noise=true , address=104.36.82.55:4198 , piece_id=MOAJGCTK2HX352AFAV2DPXQB2ABF7TGMLG34EGOBNNZV2SJIR2HA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: rpc: dial tcp 104.36.82.55:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:50.672821216 (1h56m6.453754581s) [noise=true , address=104.36.82.53:4155 , piece_id=VLRJ3PFP33L65VJBDGNDAFDC2OZLYQ3QXUFLNJZAVUWGRXWHQWLQ, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:50.672904081 (1h56m6.453837448s) [noise=true , address=77.90.31.16:4751 , piece_id=IKPXMQS4NK5SPH4OG7VVYEZPURDYOX42VNONP3NQKGVQBIASWYJA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, address:77.90.31.16:4751): protocol: write tcp 159.223.143.39:49890->77.90.31.16:4751: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:50.672923463 (1h56m6.453856845s) [seg_pos={0 0}, upload=1761]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:47:50.694930936 (1h56m6.475864306s) [upload=1761]: Done uploading segments... Jul 26 23:47:50.694956257 (1h56m6.475889633s) [upload=1761]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.026470506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.............] duration: 4.005178328s FAIL (2025-07-26 23:47:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.005211467s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X...X...X..XX.X...X..X.......X...X....XX..............X.X...] duration: 27.778145579s FAIL (2025-07-26 23:47:32 UTC) context deadline exceeded Duration: 27.778300915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 30.029294507s FAIL (2025-07-26 23:47:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 2.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595347255 (1h55m3.376280637s) [noise=true , address=104.36.82.11:4113 , piece_id=VB2HSMCNMM2NCGSDCFZCOWDXWGGJUFM2FQTDF5LNEGHFXO4RQUAA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:39866->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595370646 (1h55m3.376304037s) [seg_pos={0 0}, upload=1754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:46:47.625410173 (1h55m3.406343561s) [upload=1754]: Done uploading segments... Jul 26 23:46:47.625470564 (1h55m3.406403949s) [upload=1754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:47:01.500955539 (1h55m17.281888924s) [upload=1766]: Starting upload Jul 26 23:47:01.501081603 (1h55m17.28201498s) [upload=1766]: Done starting upload Jul 26 23:47:01.501199722 (1h55m17.2821331s) [upload=1766]: Uploading segments... Jul 26 23:47:01.501446727 (1h55m17.282380104s) [upload=1766]: Got next segment. Inline: false Jul 26 23:47:01.501522183 (1h55m17.28245556s) [upload=1766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:47:01.501530424 (1h55m17.282463802s) [upload=1766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:47:01.533216618 (1h55m17.314149999s) [seg_pos={0 0}, upload=1766]: Begin upload segment... Jul 26 23:47:01.533283292 (1h55m17.314216674s) [seg_pos={0 0}, upload=1766]: Done begin upload segment. Jul 26 23:47:01.533318241 (1h55m17.314251628s) [noise=true , address=45.91.116.12:4225 , piece_id=REQ5G3CELHPZDOCWXX4INQRJLRPHQ4T4SQ2L5HJNGPTEVF5ULOCA, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.533330156 (1h55m17.314263542s) [noise=true , address=45.91.116.8:4045 , piece_id=WJR3LN7VDQ5CXYDJRGRQGH4MF62HS3FGEHJN47PCXGBFKXN3TIVQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.533842599 (1h55m17.314775989s) [noise=true , address=45.91.116.13:4155 , piece_id=D54ANATLN425LK3BYXRWHLF5ZURXMPGRNWRE52OIPCHQ672X7F2Q, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.533912175 (1h55m17.314845541s) [noise=true , address=77.90.31.12:4590 , piece_id=EGHHI56SSVPTYOVSBDCG2ZTZM65DLXCRK6YLPWZNSGCZFDLBTEZA, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.534276158 (1h55m17.315209546s) [noise=true , address=104.36.82.7:4912 , piece_id=IYGFHYQPGE54VZLRQ4X47OE63PENODSP3TCLF6AANGU5TVGCLYUQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.534421105 (1h55m17.315354485s) [noise=true , address=45.91.116.5:4298 , piece_id=DGIB22ZRWLDPO2MYS62MGFW6RHWE4BJOUDK7FVAQK3R2UJIAVVWQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.534670997 (1h55m17.315604381s) [noise=true , address=45.91.116.6:4974 , piece_id=OEWQAOPD3IFF37256ROD4L55YA7BXLDUCGRDG5LIWJWB56BDNUQA, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.536634078 (1h55m17.317567471s) [noise=true , address=45.91.116.10:4083 , piece_id=ZXKRY4D3AJ7SFNKRRHZJ233AAW3DVMDP7FQPZVJLIIW3CVPDGDDQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.537163937 (1h55m17.318097323s) [noise=true , address=45.91.116.14:4189 , piece_id=CZRCKXYEZ6MRBAE625SSZQCDRTCZJEKXAPSRX23ATT5UOF3HAO5A, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.537649133 (1h55m17.318582519s) [noise=true , address=77.90.31.15:4699 , piece_id=2FIJ6HIPV47LF6EVSBYRPJRY2WJ6XMUZWSRCDPNQT577EULTWFMQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.538233253 (1h55m17.319166637s) [noise=true , address=45.91.116.7:4010 , piece_id=SI3QWI2QNY7WLYAGIRSVX6GD5HZZZ5GRADPZUOAYZGO3DXZ7UOOQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.538862519 (1h55m17.319795901s) [noise=true , address=104.36.82.4:4803 , piece_id=NFHH4SZHNOMJ6ZFIRS5MHPONERX4EMNO6LZIT3M25LSSH57UDXFQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.539458233 (1h55m17.320391618s) [noise=true , address=45.91.116.11:4118 , piece_id=ERMOL62UJO4CHMCZXJ4ERWATPH6TP4JBHKEOSZC663IINNOLMDYA, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.540045777 (1h55m17.320979167s) [noise=true , address=104.36.82.7:4909 , piece_id=LSPB6NGZQTTAGNU43I3Z6ICMEAYM6IY6DXTRCKW56QEJ34BM5NHA, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.540705401 (1h55m17.32163879s) [noise=true , address=45.91.116.9:4263 , piece_id=MSJNPDAO4TLUNZQA3VIOT5NNJVBHTTK7BUTXPZ3NWSPWTSXYEURQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.541339312 (1h55m17.322272716s) [noise=true , address=104.36.82.6:4874 , piece_id=GOMUVRZAQGEIKTGBLFZK5IMIKAS4MARHGN6ACXEEHTSXKAT3Z4KQ, seg_pos={0 0}, upload=1766]: Uploading piece... Jul 26 23:47:01.634889508 (1h55m17.415822882s) [upload=1766]: Next returned nil segment Jul 26 23:47:01.634907916 (1h55m17.4158413s) [upload=1766]: Waiting for error group managing segments... Jul 26 23:47:01.637150829 (1h55m17.418084191s) [noise=true , address=77.90.31.15:4699 , piece_id=2FIJ6HIPV47LF6EVSBYRPJRY2WJ6XMUZWSRCDPNQT577EULTWFMQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.637919046 (1h55m17.418852414s) [noise=true , address=45.91.116.7:4010 , piece_id=SI3QWI2QNY7WLYAGIRSVX6GD5HZZZ5GRADPZUOAYZGO3DXZ7UOOQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.638473473 (1h55m17.419406837s) [noise=true , address=45.91.116.10:4083 , piece_id=ZXKRY4D3AJ7SFNKRRHZJ233AAW3DVMDP7FQPZVJLIIW3CVPDGDDQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.639102724 (1h55m17.420036091s) [noise=true , address=45.91.116.11:4118 , piece_id=ERMOL62UJO4CHMCZXJ4ERWATPH6TP4JBHKEOSZC663IINNOLMDYA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.639769712 (1h55m17.420703075s) [noise=true , address=45.91.116.8:4045 , piece_id=WJR3LN7VDQ5CXYDJRGRQGH4MF62HS3FGEHJN47PCXGBFKXN3TIVQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.640400722 (1h55m17.421334092s) [noise=true , address=45.91.116.6:4974 , piece_id=OEWQAOPD3IFF37256ROD4L55YA7BXLDUCGRDG5LIWJWB56BDNUQA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.640867399 (1h55m17.42180077s) [noise=true , address=45.91.116.12:4225 , piece_id=REQ5G3CELHPZDOCWXX4INQRJLRPHQ4T4SQ2L5HJNGPTEVF5ULOCA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.641311046 (1h55m17.42224442s) [noise=true , address=45.91.116.14:4189 , piece_id=CZRCKXYEZ6MRBAE625SSZQCDRTCZJEKXAPSRX23ATT5UOF3HAO5A, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.641798758 (1h55m17.422732148s) [noise=true , address=45.91.116.13:4155 , piece_id=D54ANATLN425LK3BYXRWHLF5ZURXMPGRNWRE52OIPCHQ672X7F2Q, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.761950372 (1h55m17.542883754s) [noise=true , address=45.91.116.9:4263 , piece_id=MSJNPDAO4TLUNZQA3VIOT5NNJVBHTTK7BUTXPZ3NWSPWTSXYEURQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:01.778401865 (1h55m17.559335235s) [noise=true , address=45.91.116.5:4298 , piece_id=DGIB22ZRWLDPO2MYS62MGFW6RHWE4BJOUDK7FVAQK3R2UJIAVVWQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:03.029323184 (1h55m18.810256565s) [noise=true , address=77.90.31.12:4590 , piece_id=EGHHI56SSVPTYOVSBDCG2ZTZM65DLXCRK6YLPWZNSGCZFDLBTEZA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil> Jul 26 23:47:31.471281253 (1h55m47.252214628s) [noise=true , address=104.36.82.7:4912 , piece_id=IYGFHYQPGE54VZLRQ4X47OE63PENODSP3TCLF6AANGU5TVGCLYUQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:31.471351103 (1h55m47.252284525s) [noise=true , address=104.36.82.4:4803 , piece_id=NFHH4SZHNOMJ6ZFIRS5MHPONERX4EMNO6LZIT3M25LSSH57UDXFQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 159.223.143.39:37766->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:31.471383566 (1h55m47.252316935s) [noise=true , address=104.36.82.7:4909 , piece_id=LSPB6NGZQTTAGNU43I3Z6ICMEAYM6IY6DXTRCKW56QEJ34BM5NHA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 159.223.143.39:39038->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:31.471421220 (1h55m47.252354596s) [noise=true , address=104.36.82.6:4874 , piece_id=GOMUVRZAQGEIKTGBLFZK5IMIKAS4MARHGN6ACXEEHTSXKAT3Z4KQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:31.471469597 (1h55m47.252402965s) [seg_pos={0 0}, upload=1766]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:47:31.498930488 (1h55m47.279863862s) [upload=1766]: Done uploading segments... Jul 26 23:47:31.498966456 (1h55m47.279899821s) [upload=1766]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.029322092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [.X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 862.465011ms FAIL (2025-07-26 23:47:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 862.48758ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [..X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.02723166s FAIL (2025-07-26 23:47:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.027258108s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XX...X...X..XX.X...X..X.......X...X....XX..............X.X..] duration: 21.26828273s FAIL (2025-07-26 23:47:04 UTC) context deadline exceeded Duration: 21.268317173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......XXX.X....XX.X.X.X.......XXX...XX.....X.XXX............] duration: 1.94635662s FAIL (2025-07-26 23:46:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.94640006s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 30.039771731s FAIL (2025-07-26 23:46:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s={0 0}, upload=1747]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:46:17.524551936 (1h54m33.305485313s) [upload=1747]: Done uploading segments... Jul 26 23:46:17.524598794 (1h54m33.30553217s) [upload=1747]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:46:17.624596775 (1h54m33.40553016s) [upload=1754]: Starting upload Jul 26 23:46:17.624681788 (1h54m33.405615175s) [upload=1754]: Done starting upload Jul 26 23:46:17.625039922 (1h54m33.405973284s) [upload=1754]: Uploading segments... Jul 26 23:46:17.625108200 (1h54m33.406041562s) [upload=1754]: Got next segment. Inline: false Jul 26 23:46:17.625149000 (1h54m33.406082386s) [upload=1754]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:46:17.625155538 (1h54m33.406088925s) [upload=1754]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:46:17.655075907 (1h54m33.436009295s) [seg_pos={0 0}, upload=1754]: Begin upload segment... Jul 26 23:46:17.655137255 (1h54m33.43607085s) [seg_pos={0 0}, upload=1754]: Done begin upload segment. Jul 26 23:46:17.655170740 (1h54m33.436104127s) [noise=true , address=104.36.82.11:4115 , piece_id=7PO7SY2IFSUSKGCFSTEYHB7RDT26Q525P6IN5X5CFW53ZXPUHRDQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.655597579 (1h54m33.436530958s) [noise=true , address=45.91.116.7:4009 , piece_id=PDAZ2CTPP3LACPCZ4NPIAB2MAIIOB274NOS7YQX72VEJ5UMXN6GQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.655984837 (1h54m33.436918224s) [noise=true , address=45.91.116.10:4082 , piece_id=UOTJQFI33SQPSQZMNSLETPXE4JCW4WDJ4J334VQKUA425R2JAAAQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.656528096 (1h54m33.437461475s) [noise=true , address=45.91.116.4:4940 , piece_id=GDHFUEEC7D3Y6A4MMRBJWRTNJXRJ2JWGJFSGV575XHCRAHGUPIMA, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.656879091 (1h54m33.437812474s) [noise=true , address=104.36.82.14:4223 , piece_id=EOFC4DVGIVEC4YMI3WI4JCKCMGQPPEGZ6LBC76FZTMAMLQ7VF6IQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.657222660 (1h54m33.438156046s) [noise=true , address=104.36.82.3:4767 , piece_id=6H3PKDYKCPSGYZN5YV75ZTTV2FLOQYXI3RFO5HKH5QIZ74QFL3HQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.657578618 (1h54m33.438512005s) [noise=true , address=45.91.116.7:4010 , piece_id=4EGNACS2CUB2TDRFFXEGGPT4O4CUCPGZYTHMMRBBNGUK46QW4GRA, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.657902329 (1h54m33.438835713s) [noise=true , address=104.36.82.6:4874 , piece_id=TTCCDLXB3Y7J4OMKFSWEHH762ZRFQTDO7ULTM647HIOC4VZFE3VQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.658191074 (1h54m33.439124454s) [noise=true , address=104.36.82.5:4838 , piece_id=NCUDT2LX5WWLIAVEC66NBYAXROJGYAM2UYP5TNQPK6FSO4CWKPNQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.658492019 (1h54m33.439425405s) [noise=true , address=104.36.82.7:4912 , piece_id=E26RTIRTXF6AE4FD4QYSB4BDZAA7AH4MVB56QMSJAM46EVO6S5AQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.658782220 (1h54m33.4397156s) [noise=true , address=104.36.82.11:4113 , piece_id=VB2HSMCNMM2NCGSDCFZCOWDXWGGJUFM2FQTDF5LNEGHFXO4RQUAA, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.659321098 (1h54m33.440254475s) [noise=true , address=45.91.116.8:4047 , piece_id=ONF5J6732S6FLJMYO6A4MKQEIDTBOWBE2UZ5BCVXY56A7GZ64PIQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.659765533 (1h54m33.440698909s) [noise=true , address=45.91.116.5:4297 , piece_id=HQK3FUYO7SFJGQN33ADHIICOZBLUKSCCH42DGATPAFVNEVVF3GGQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.660232930 (1h54m33.441166306s) [noise=true , address=45.91.116.10:4083 , piece_id=4XZCNJRB7XPPQJFZDICZVLO5KNN7G7Z44S42HUBLXRG2NZUYDB5Q, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.660638792 (1h54m33.441572178s) [noise=true , address=77.90.31.10:4518 , piece_id=MEMHKZDGOL5F4WDBQNOBIUGVK33PQBH7CQZRTGOPHHKXBNVDR3NA, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.661014017 (1h54m33.441947393s) [noise=true , address=45.91.116.12:4225 , piece_id=N3KUTSOCDBJ73CS7XIJMS2NJFM2S74ZQ344MIRT3QSOT6BHP6NAQ, seg_pos={0 0}, upload=1754]: Uploading piece... Jul 26 23:46:17.742771369 (1h54m33.523704749s) [upload=1754]: Next returned nil segment Jul 26 23:46:17.742810540 (1h54m33.523743904s) [upload=1754]: Waiting for error group managing segments... Jul 26 23:46:17.753482767 (1h54m33.53441613s) [noise=true , address=45.91.116.7:4009 , piece_id=PDAZ2CTPP3LACPCZ4NPIAB2MAIIOB274NOS7YQX72VEJ5UMXN6GQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:17.753537613 (1h54m33.534470981s) [noise=true , address=45.91.116.10:4083 , piece_id=4XZCNJRB7XPPQJFZDICZVLO5KNN7G7Z44S42HUBLXRG2NZUYDB5Q, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:17.754145497 (1h54m33.535078869s) [noise=true , address=45.91.116.8:4047 , piece_id=ONF5J6732S6FLJMYO6A4MKQEIDTBOWBE2UZ5BCVXY56A7GZ64PIQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:17.754171756 (1h54m33.535105188s) [noise=true , address=77.90.31.10:4518 , piece_id=MEMHKZDGOL5F4WDBQNOBIUGVK33PQBH7CQZRTGOPHHKXBNVDR3NA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:17.754601308 (1h54m33.53553468s) [noise=true , address=45.91.116.10:4082 , piece_id=UOTJQFI33SQPSQZMNSLETPXE4JCW4WDJ4J334VQKUA425R2JAAAQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:17.755125717 (1h54m33.53605909s) [noise=true , address=45.91.116.4:4940 , piece_id=GDHFUEEC7D3Y6A4MMRBJWRTNJXRJ2JWGJFSGV575XHCRAHGUPIMA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:17.755155528 (1h54m33.536088895s) [noise=true , address=45.91.116.12:4225 , piece_id=N3KUTSOCDBJ73CS7XIJMS2NJFM2S74ZQ344MIRT3QSOT6BHP6NAQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:17.755212459 (1h54m33.536145837s) [noise=true , address=45.91.116.7:4010 , piece_id=4EGNACS2CUB2TDRFFXEGGPT4O4CUCPGZYTHMMRBBNGUK46QW4GRA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:17.908402062 (1h54m33.689335425s) [noise=true , address=45.91.116.5:4297 , piece_id=HQK3FUYO7SFJGQN33ADHIICOZBLUKSCCH42DGATPAFVNEVVF3GGQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil> Jul 26 23:46:47.594360102 (1h55m3.375293486s) [noise=true , address=104.36.82.6:4874 , piece_id=TTCCDLXB3Y7J4OMKFSWEHH762ZRFQTDO7ULTM647HIOC4VZFE3VQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:48862->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595112735 (1h55m3.376046115s) [noise=true , address=104.36.82.7:4912 , piece_id=E26RTIRTXF6AE4FD4QYSB4BDZAA7AH4MVB56QMSJAM46EVO6S5AQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595193825 (1h55m3.376127208s) [noise=true , address=104.36.82.11:4115 , piece_id=7PO7SY2IFSUSKGCFSTEYHB7RDT26Q525P6IN5X5CFW53ZXPUHRDQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595235841 (1h55m3.376169219s) [noise=true , address=104.36.82.14:4223 , piece_id=EOFC4DVGIVEC4YMI3WI4JCKCMGQPPEGZ6LBC76FZTMAMLQ7VF6IQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595269175 (1h55m3.376202558s) [noise=true , address=104.36.82.3:4767 , piece_id=6H3PKDYKCPSGYZN5YV75ZTTV2FLOQYXI3RFO5HKH5QIZ74QFL3HQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595302169 (1h55m3.376235549s) [noise=true , address=104.36.82.5:4838 , piece_id=NCUDT2LX5WWLIAVEC66NBYAXROJGYAM2UYP5TNQPK6FSO4CWKPNQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595347255 (1h55m3.376280637s) [noise=true , address=104.36.82.11:4113 , piece_id=VB2HSMCNMM2NCGSDCFZCOWDXWGGJUFM2FQTDF5LNEGHFXO4RQUAA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:39866->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:47.595370646 (1h55m3.376304037s) [seg_pos={0 0}, upload=1754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:46:47.625410173 (1h55m3.406343561s) [upload=1754]: Done uploading segments... Jul 26 23:46:47.625470564 (1h55m3.406403949s) [upload=1754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.039797328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033872509s FAIL (2025-07-26 23:46:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== , piece_id=XPCWZYOKZAFB3BA6NDWY6RCIA3THYOCCRAOB7UINHCASKAVC37KQ, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.571912330 (1h53m54.352845712s) [noise=true , address=79.127.201.215:4355 , piece_id=LOREMUNEAOQ3ASXQLU4Q7ISP2LFVIINIGKF2Y54YIEA4XPMJ627A, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.572544233 (1h53m54.353477616s) [noise=true , address=104.36.82.47:4536 , piece_id=SR27WNQKZMNV53T65EA6ZDCQQTDL42NQ7NPIF3DU4DTFKXBHHIDA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.572966807 (1h53m54.353900199s) [noise=true , address=77.90.31.17:4799 , piece_id=FYMB36VLWYJVN3XAIZBWB5KH4DJDJNIEUQDHCEGKZ5FGKB4NCDRA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.573391378 (1h53m54.354324756s) [noise=true , address=104.36.82.56:4229 , piece_id=D3ITSMS7GM4NCMQV44JFE2IGVN5XI7SDAVNHRG6I6BYEDF5KXLZA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.573845783 (1h53m54.35477916s) [noise=true , address=79.127.219.38:4004 , piece_id=7TVBMQKTFI554JKMI4WJJ2T4C4AZ3IVTDNTJZHBALPARD7JG4MEQ, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.574267669 (1h53m54.355201046s) [noise=true , address=104.36.82.46:4509 , piece_id=6QEQM3Y6IU5Y3CHGYFHSC4652TGPYEB6Q6GG4RT6OZHGLM6HEMKQ, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.574709734 (1h53m54.35564312s) [noise=true , address=104.36.82.53:4128 , piece_id=BTLRM2F7OZ2NZZCQJZXT2XJ2GF46YKHYXRVU4XOOZASTLO3YYZXA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.575215208 (1h53m54.356148594s) [noise=true , address=64.49.13.140:4508 , piece_id=XV6DTKPFOHFO5765LYJ2PUMIEDBAQHO37BCOMEL5UVHK7OA6DYZA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.575682258 (1h53m54.35661564s) [noise=true , address=79.127.205.246:4280 , piece_id=EVFHHFB7SBQFAP64QR56QU3X2KBMPCGNDTF6TCMN5EO6NQJKMXNA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.576219733 (1h53m54.357153121s) [noise=true , address=79.127.220.97:4361 , piece_id=FTWC5PMPCTOUGNBWTFT6HVGSQ25CCWIXHVPOA246XOZPEBMEHP6A, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.576719265 (1h53m54.357652644s) [noise=true , address=77.90.31.21:4895 , piece_id=GIO2QXWPRLEB2HSEPASEUVLU4A3OKZIH4AME4Q5XMLPYUT6RMUAA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.577249070 (1h53m54.358182456s) [noise=true , address=79.127.205.235:4687 , piece_id=DDSOVQH2T44BYM2FEFSO6EU4NDN3FJ3PU3D57SKEIFDUK2WV5YPA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.580325096 (1h53m54.361258487s) [noise=true , address=77.90.31.15:4715 , piece_id=EAEXN7IZRW5KKXOLUPMGML3GXMIYJTOORCQA4TGJ7QYFD2ZOJU3A, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.580921670 (1h53m54.36185506s) [noise=true , address=104.36.82.54:4175 , piece_id=HIYPJX4N7N5K6FLVYA4327IWS7M55BUSSJKELUV7NG2IWTSOZKVA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.581391842 (1h53m54.362325209s) [noise=true , address=64.49.13.138:4483 , piece_id=TZAKN36VPGXHMV5JTZ7GPX2FSY6XP2SEQ5UPFADYDFU5BE7YSNZQ, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.582279540 (1h53m54.363212928s) [noise=true , address=104.36.82.20:4440 , piece_id=VJOTKS3DPZCFVNK6AOY2NH32GWX3BMKDRDNPLVWKZRJX7HWRTTYA, seg_pos={0 0}, upload=1745]: Uploading piece... Jul 26 23:45:38.709721926 (1h53m54.49065531s) [upload=1745]: Next returned nil segment Jul 26 23:45:38.709742350 (1h53m54.490675716s) [upload=1745]: Waiting for error group managing segments... Jul 26 23:45:38.717073278 (1h53m54.498006649s) [noise=true , address=79.127.219.38:4004 , piece_id=7TVBMQKTFI554JKMI4WJJ2T4C4AZ3IVTDNTJZHBALPARD7JG4MEQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:45:38.753237211 (1h53m54.534170581s) [noise=true , address=77.90.31.17:4799 , piece_id=FYMB36VLWYJVN3XAIZBWB5KH4DJDJNIEUQDHCEGKZ5FGKB4NCDRA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:45:38.756214249 (1h53m54.537147611s) [noise=true , address=77.90.31.21:4895 , piece_id=GIO2QXWPRLEB2HSEPASEUVLU4A3OKZIH4AME4Q5XMLPYUT6RMUAA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:45:38.787695172 (1h53m54.568628534s) [noise=true , address=77.90.31.15:4715 , piece_id=EAEXN7IZRW5KKXOLUPMGML3GXMIYJTOORCQA4TGJ7QYFD2ZOJU3A, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:45:39.210707133 (1h53m54.991640495s) [noise=true , address=79.127.205.246:4280 , piece_id=EVFHHFB7SBQFAP64QR56QU3X2KBMPCGNDTF6TCMN5EO6NQJKMXNA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:45:40.003715113 (1h53m55.784648494s) [noise=true , address=64.49.13.140:4508 , piece_id=XV6DTKPFOHFO5765LYJ2PUMIEDBAQHO37BCOMEL5UVHK7OA6DYZA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:45:40.323250495 (1h53m56.104183865s) [noise=true , address=79.127.205.235:4687 , piece_id=DDSOVQH2T44BYM2FEFSO6EU4NDN3FJ3PU3D57SKEIFDUK2WV5YPA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:45:41.328958532 (1h53m57.109891907s) [noise=true , address=79.127.220.97:4361 , piece_id=FTWC5PMPCTOUGNBWTFT6HVGSQ25CCWIXHVPOA246XOZPEBMEHP6A, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:45:42.814520097 (1h53m58.595453478s) [noise=true , address=64.49.13.138:4483 , piece_id=TZAKN36VPGXHMV5JTZ7GPX2FSY6XP2SEQ5UPFADYDFU5BE7YSNZQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:46:19.496410067 (1h54m35.277343429s) [noise=true , address=79.127.201.215:4355 , piece_id=LOREMUNEAOQ3ASXQLU4Q7ISP2LFVIINIGKF2Y54YIEA4XPMJ627A, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil> Jul 26 23:46:38.468888642 (1h54m54.24982201s) [noise=true , address=104.36.82.54:4175 , piece_id=HIYPJX4N7N5K6FLVYA4327IWS7M55BUSSJKELUV7NG2IWTSOZKVA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:46:38.469217903 (1h54m54.250151384s) [noise=true , address=104.36.82.55:4194 , piece_id=XPCWZYOKZAFB3BA6NDWY6RCIA3THYOCCRAOB7UINHCASKAVC37KQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: write tcp 159.223.143.39:33582->104.36.82.55:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:38.469333760 (1h54m54.250267151s) [noise=true , address=104.36.82.46:4509 , piece_id=6QEQM3Y6IU5Y3CHGYFHSC4652TGPYEB6Q6GG4RT6OZHGLM6HEMKQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: write tcp 159.223.143.39:52422->104.36.82.46:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:38.469490069 (1h54m54.250423432s) [noise=true , address=104.36.82.53:4128 , piece_id=BTLRM2F7OZ2NZZCQJZXT2XJ2GF46YKHYXRVU4XOOZASTLO3YYZXA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 159.223.143.39:53512->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:38.469648942 (1h54m54.250582307s) [noise=true , address=104.36.82.20:4457 , piece_id=77AJOFAEB4EAG7C7TVLZFATTRBLQABPZVJ2TQU2LEY6FGAFFCSQQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:16sDrSF9mwW9cQ2h8N5TfUKrnwPAwtEfed6AwCEAT95r5iY9vQ, address:104.36.82.20:4457): protocol: write tcp 159.223.143.39:60032->104.36.82.20:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:38.469734893 (1h54m54.250668273s) [noise=true , address=104.36.82.47:4536 , piece_id=SR27WNQKZMNV53T65EA6ZDCQQTDL42NQ7NPIF3DU4DTFKXBHHIDA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: write tcp 159.223.143.39:53182->104.36.82.47:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:38.469870044 (1h54m54.250803421s) [noise=true , address=104.36.82.56:4229 , piece_id=D3ITSMS7GM4NCMQV44JFE2IGVN5XI7SDAVNHRG6I6BYEDF5KXLZA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: write tcp 159.223.143.39:50170->104.36.82.56:4229: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:38.469894808 (1h54m54.25082819s) [noise=true , address=104.36.82.20:4440 , piece_id=VJOTKS3DPZCFVNK6AOY2NH32GWX3BMKDRDNPLVWKZRJX7HWRTTYA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:104.36.82.20:4440): protocol: write tcp 159.223.143.39:52280->104.36.82.20:4440: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:38.469935760 (1h54m54.250869146s) [seg_pos={0 0}, upload=1745]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:46:38.500714241 (1h54m54.281647611s) [upload=1745]: Done uploading segments... Jul 26 23:46:38.500770670 (1h54m54.281704048s) [upload=1745]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.033918276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [...X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.070774948s FAIL (2025-07-26 23:46:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.070813611s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...........] duration: 1.840566776s FAIL (2025-07-26 23:46:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.840591407s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 30.027787578s FAIL (2025-07-26 23:46:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 24.191083224s) [upload=1728]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:45:47.528550027 (1h54m3.309483396s) [upload=1747]: Starting upload Jul 26 23:45:47.528640839 (1h54m3.309574218s) [upload=1747]: Done starting upload Jul 26 23:45:47.528745650 (1h54m3.309679015s) [upload=1747]: Uploading segments... Jul 26 23:45:47.529023904 (1h54m3.309957269s) [upload=1747]: Got next segment. Inline: false Jul 26 23:45:47.529113546 (1h54m3.310046926s) [upload=1747]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:45:47.529121441 (1h54m3.31005482s) [upload=1747]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:45:47.558220849 (1h54m3.339154215s) [seg_pos={0 0}, upload=1747]: Begin upload segment... Jul 26 23:45:47.558283441 (1h54m3.33921682s) [seg_pos={0 0}, upload=1747]: Done begin upload segment. Jul 26 23:45:47.558351476 (1h54m3.339284867s) [noise=true , address=104.36.82.11:4115 , piece_id=TFH32Q4SWNEBDGDUTIIBRF23IZIIR3HBPKJ2XXBECEL2ZAYJGEDQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.558369352 (1h54m3.339302731s) [noise=true , address=104.36.82.13:4187 , piece_id=62DZZ7VXJMTOEDKM3X4PCC62RT7DZFIYBAKJITPFJ33E62M4RTVQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.558827957 (1h54m3.339761335s) [noise=true , address=77.90.31.19:4842 , piece_id=4XPIOROXEUXWGFSSGILHJQUHEMSOBZWC7OM3YK5NKQTJ7UYJIHTQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.558903236 (1h54m3.33983662s) [noise=true , address=45.91.116.13:4154 , piece_id=5KPL33WBEOW26RPXI4ZO4UBD26NAIYM24L3WT2TE35NHZRGZXZVQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.559214896 (1h54m3.34014828s) [noise=true , address=104.36.82.12:4151 , piece_id=BYQLIFBJM2SV2GRDQSVX2DCBY5EDJY4ZOSPHFV7Y4QWF74QJMURA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.559661333 (1h54m3.340594713s) [noise=true , address=45.91.116.12:4227 , piece_id=YAB7MCF4WHAYQUB7ORLXZRBGMKBJ4X3XPQUH26ULZTWQH7IY6KMQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.559999040 (1h54m3.340932417s) [noise=true , address=45.91.116.8:4047 , piece_id=57T5JDPBGBYXXLEHQPXWZ2QWNHL7MJQUIJ52CDKGCCRCOTB2MLRA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560036872 (1h54m3.340970257s) [noise=true , address=104.36.82.11:4114 , piece_id=MMP6F2OP57H57YPYHJVHKOTDKMPGW2UJL6GSN7VKTBYXWN2YHV7Q, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560402073 (1h54m3.341335454s) [noise=true , address=104.36.82.15:4259 , piece_id=PG6G724ZDDZ7KUTJQGFEKWI6BLABYPOD4ZC3MUYNN5JKIIMCC2SA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560498278 (1h54m3.341431655s) [noise=true , address=104.36.82.14:4221 , piece_id=SLEEOKCQPOVD656KFD7QNFVYF4ZFVBKJYYMVFRDILH6Q3HSTKBVQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560707268 (1h54m3.341640635s) [noise=true , address=104.36.82.12:4149 , piece_id=T6UFFEOQVPWMUJVBIW5C2ZMQOLNCS2QE4IL7KRFMRDYJ6J2WNE5A, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.560776189 (1h54m3.341709568s) [noise=true , address=77.90.31.13:4627 , piece_id=YSSOPU5BKDLVEDTNGYJLPOC3YH4NWNMLQCXEVGVXMNTB6E3EB2QA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.561074124 (1h54m3.342007503s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6QBRXDNYS56E5ZE37TWO3DNBQHKTSBME3IO2OQ6M3MCH2LBR4BA, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.561367507 (1h54m3.342300882s) [noise=true , address=45.91.116.6:4975 , piece_id=BSA3QO43FCGA7D6F74TXTFCSHSG7RXC7KZCUPCNLLEFMNHQ4TGHQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.561671774 (1h54m3.342605152s) [noise=true , address=45.91.116.5:4297 , piece_id=ZV6TFLM2X7VASLP2GEME7G7NYWNRLYYMWMT7IZRK5IEWHPAHJ3MQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.561996146 (1h54m3.342929522s) [noise=true , address=104.36.82.7:4909 , piece_id=WSHEAAFVGA63SPALM2XPNSNNPSRZPKK2J4VTP3PT3XOYBTFP6LSQ, seg_pos={0 0}, upload=1747]: Uploading piece... Jul 26 23:45:47.649771810 (1h54m3.430705175s) [upload=1747]: Next returned nil segment Jul 26 23:45:47.649792842 (1h54m3.430726211s) [upload=1747]: Waiting for error group managing segments... Jul 26 23:45:47.653651879 (1h54m3.434585247s) [noise=true , address=45.91.116.6:4975 , piece_id=BSA3QO43FCGA7D6F74TXTFCSHSG7RXC7KZCUPCNLLEFMNHQ4TGHQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.655490460 (1h54m3.436423829s) [noise=true , address=45.91.116.8:4047 , piece_id=57T5JDPBGBYXXLEHQPXWZ2QWNHL7MJQUIJ52CDKGCCRCOTB2MLRA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.656316568 (1h54m3.437249937s) [noise=true , address=45.91.116.13:4154 , piece_id=5KPL33WBEOW26RPXI4ZO4UBD26NAIYM24L3WT2TE35NHZRGZXZVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.658051626 (1h54m3.438984993s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6QBRXDNYS56E5ZE37TWO3DNBQHKTSBME3IO2OQ6M3MCH2LBR4BA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.663908347 (1h54m3.444841723s) [noise=true , address=45.91.116.12:4227 , piece_id=YAB7MCF4WHAYQUB7ORLXZRBGMKBJ4X3XPQUH26ULZTWQH7IY6KMQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:47.889974750 (1h54m3.67090812s) [noise=true , address=45.91.116.5:4297 , piece_id=ZV6TFLM2X7VASLP2GEME7G7NYWNRLYYMWMT7IZRK5IEWHPAHJ3MQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:49.022796696 (1h54m4.803730077s) [noise=true , address=77.90.31.13:4627 , piece_id=YSSOPU5BKDLVEDTNGYJLPOC3YH4NWNMLQCXEVGVXMNTB6E3EB2QA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:45:49.105919749 (1h54m4.886853117s) [noise=true , address=77.90.31.19:4842 , piece_id=4XPIOROXEUXWGFSSGILHJQUHEMSOBZWC7OM3YK5NKQTJ7UYJIHTQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil> Jul 26 23:46:17.498482600 (1h54m33.279415977s) [noise=true , address=104.36.82.12:4149 , piece_id=T6UFFEOQVPWMUJVBIW5C2ZMQOLNCS2QE4IL7KRFMRDYJ6J2WNE5A, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498555419 (1h54m33.279488797s) [noise=true , address=104.36.82.13:4187 , piece_id=62DZZ7VXJMTOEDKM3X4PCC62RT7DZFIYBAKJITPFJ33E62M4RTVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498588278 (1h54m33.279521641s) [noise=true , address=104.36.82.12:4151 , piece_id=BYQLIFBJM2SV2GRDQSVX2DCBY5EDJY4ZOSPHFV7Y4QWF74QJMURA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498659084 (1h54m33.27959247s) [noise=true , address=104.36.82.11:4115 , piece_id=TFH32Q4SWNEBDGDUTIIBRF23IZIIR3HBPKJ2XXBECEL2ZAYJGEDQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498675455 (1h54m33.279608817s) [noise=true , address=104.36.82.11:4114 , piece_id=MMP6F2OP57H57YPYHJVHKOTDKMPGW2UJL6GSN7VKTBYXWN2YHV7Q, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:47826->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498689802 (1h54m33.279623172s) [noise=true , address=104.36.82.7:4909 , piece_id=WSHEAAFVGA63SPALM2XPNSNNPSRZPKK2J4VTP3PT3XOYBTFP6LSQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498733751 (1h54m33.279667257s) [noise=true , address=104.36.82.15:4259 , piece_id=PG6G724ZDDZ7KUTJQGFEKWI6BLABYPOD4ZC3MUYNN5JKIIMCC2SA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498783558 (1h54m33.279716935s) [noise=true , address=104.36.82.14:4221 , piece_id=SLEEOKCQPOVD656KFD7QNFVYF4ZFVBKJYYMVFRDILH6Q3HSTKBVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:33490->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:17.498804857 (1h54m33.279738224s) [seg_pos={0 0}, upload=1747]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:46:17.524551936 (1h54m33.305485313s) [upload=1747]: Done uploading segments... Jul 26 23:46:17.524598794 (1h54m33.30553217s) [upload=1747]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.027876714s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX..........] duration: 2.834682599s FAIL (2025-07-26 23:46:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.834709429s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.XX...X...X..XX.X...X..X.......X...X....XX..............X.X.] duration: 19.422835539s FAIL (2025-07-26 23:46:00 UTC) context deadline exceeded Duration: 19.422862753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [....X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.034225081s FAIL (2025-07-26 23:45:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.034250318s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..XX...X...X..XX.X...X..X.......X...X....XX..............X.X] duration: 1m0.001079875s FAIL (2025-07-26 23:45:41 UTC) context deadline exceeded Duration: 1m0.001168071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03371062s FAIL (2025-07-26 23:45:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=1726]: Done begin upload segment. Jul 26 23:44:35.488979313 (1h52m51.269912699s) [noise=true , address=104.36.82.53:4149 , piece_id=5I4OY4D4WBZA3FB7HLKCPWDQAI7TKPFDP3IECR5J6TVHQ7HQ2K5Q, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.489001662 (1h52m51.269935044s) [noise=true , address=79.127.205.235:4691 , piece_id=34QHDCOHKOGY6YAAX5C2U44STARV6AQJFODJB4WTPGLWPQ47J56A, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.489541468 (1h52m51.270474856s) [noise=true , address=64.49.13.140:4508 , piece_id=KXAO5GNBNBWWQ3KMC4VIE5KAVIXATL3DBVZM4EUAPUKUKXF2WRGA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.489575066 (1h52m51.270508451s) [noise=true , address=77.90.31.21:4912 , piece_id=PIA7P2VPFVLBH4GFYRHQ5KP5PRSCJCDOYXJYNZEDGKDYHULGBUBQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.490050535 (1h52m51.270983919s) [noise=true , address=104.36.82.46:4490 , piece_id=QFKKQU7UEDERMIXEIWGAEAUMRCGUANOC5ZMM7HOWUBFBFZL6EQIA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.490125109 (1h52m51.271058494s) [noise=true , address=104.36.82.56:4247 , piece_id=2FDHHAMOQO6RGUFXLQH4UKNHYPOFYZTPEABXDHST6RVWJC5CJYCA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.490686708 (1h52m51.271620098s) [noise=true , address=64.49.13.138:4482 , piece_id=6KPYSZAM4AJTFBVHDNLPJY7CGLGJSIKPB7S7N4G5F4ZIRE4G3SGQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.490794891 (1h52m51.271728282s) [noise=true , address=64.49.13.139:4478 , piece_id=RQBUGIMT7XHLF2FBZGUA4RTRYUN6FL45RIZH2YEUFBKPLBR7P6UA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.491316121 (1h52m51.272249506s) [noise=true , address=156.146.43.225:4448 , piece_id=LIAN4KFQUGNFOX4PLKNJ6GHICUDTCEIFRIULLRHYENJUYPR4TF2Q, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.491691604 (1h52m51.272624987s) [noise=true , address=156.146.43.225:4449 , piece_id=ZVXMQF7BR7FMGBFFFF5UPAPACIOIWNOUK3JCOG62VOG4D24FC3TQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.492241107 (1h52m51.273174493s) [noise=true , address=77.90.31.11:4564 , piece_id=W4XJ3TSB5PXNGV2ZIASXSVNVOHFH4SEPUE3MBWWSGINNPLXEFP2Q, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.492725186 (1h52m51.273658573s) [noise=true , address=104.36.82.52:4107 , piece_id=DQKOQZXURQQDTVR6ELRV5CRIFXFWJFSIMHCIJSGKDXZKV5A6C7PA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.493214294 (1h52m51.274147682s) [noise=true , address=77.90.31.19:4871 , piece_id=7ZSMZ2RIR42ZVPSFQ3ILXTGA4LTSUIBTMINIGLFHWSD6YCH4ASRA, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.494016913 (1h52m51.274950301s) [noise=true , address=79.127.219.38:4001 , piece_id=4VIS7HGYCK4JHUIZHCRIKNI4A6UZEF6VE6Z2TPVYS7E2WOJYCVRQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.494443652 (1h52m51.275377043s) [noise=true , address=77.90.31.12:4608 , piece_id=27LCIXMJX4LBTBANJPOMVNA2W3LC6LRFWMM4PYS2PPU3XFUE7NNQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.495095329 (1h52m51.276028718s) [noise=true , address=104.36.82.13:4200 , piece_id=HSLZPW5YURREZAGGW5LQN7QTDIKB4WX54IDJI4MFHMV3WILAC6NQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.495535278 (1h52m51.276468675s) [noise=true , address=104.36.82.55:4199 , piece_id=RDOM7TE4TMKCIJF7CWLO6L2HOO7HNTEOLEWK3YLXXJ263CEZP4GQ, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.495982709 (1h52m51.276916088s) [noise=true , address=104.36.82.47:4522 , piece_id=VMPSGRVEATIN2IKU7MBPJTWW4DWKKGOWDBWYSLER7T3J3HDMPB3A, seg_pos={0 0}, upload=1726]: Uploading piece... Jul 26 23:44:35.565482117 (1h52m51.346415485s) [upload=1726]: Next returned nil segment Jul 26 23:44:35.565499405 (1h52m51.346432784s) [upload=1726]: Waiting for error group managing segments... Jul 26 23:44:35.570357164 (1h52m51.351290541s) [noise=true , address=79.127.219.38:4001 , piece_id=4VIS7HGYCK4JHUIZHCRIKNI4A6UZEF6VE6Z2TPVYS7E2WOJYCVRQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:35.589069819 (1h52m51.370003196s) [noise=true , address=77.90.31.12:4608 , piece_id=27LCIXMJX4LBTBANJPOMVNA2W3LC6LRFWMM4PYS2PPU3XFUE7NNQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:35.615620249 (1h52m51.396553628s) [noise=true , address=77.90.31.21:4912 , piece_id=PIA7P2VPFVLBH4GFYRHQ5KP5PRSCJCDOYXJYNZEDGKDYHULGBUBQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:35.666362835 (1h52m51.447296203s) [noise=true , address=77.90.31.11:4564 , piece_id=W4XJ3TSB5PXNGV2ZIASXSVNVOHFH4SEPUE3MBWWSGINNPLXEFP2Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:35.843127855 (1h52m51.624061235s) [noise=true , address=64.49.13.139:4478 , piece_id=RQBUGIMT7XHLF2FBZGUA4RTRYUN6FL45RIZH2YEUFBKPLBR7P6UA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000 Jul 26 23:44:36.095341527 (1h52m51.876274889s) [noise=true , address=79.127.205.235:4691 , piece_id=34QHDCOHKOGY6YAAX5C2U44STARV6AQJFODJB4WTPGLWPQ47J56A, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:36.619796257 (1h52m52.400729624s) [noise=true , address=64.49.13.140:4508 , piece_id=KXAO5GNBNBWWQ3KMC4VIE5KAVIXATL3DBVZM4EUAPUKUKXF2WRGA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:44:37.623736203 (1h52m53.404669582s) [noise=true , address=156.146.43.225:4448 , piece_id=LIAN4KFQUGNFOX4PLKNJ6GHICUDTCEIFRIULLRHYENJUYPR4TF2Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:38.346038855 (1h52m54.126972234s) [noise=true , address=64.49.13.138:4482 , piece_id=6KPYSZAM4AJTFBVHDNLPJY7CGLGJSIKPB7S7N4G5F4ZIRE4G3SGQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:38.748590410 (1h52m54.529523779s) [noise=true , address=77.90.31.19:4871 , piece_id=7ZSMZ2RIR42ZVPSFQ3ILXTGA4LTSUIBTMINIGLFHWSD6YCH4ASRA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:44:39.757730680 (1h52m55.538664049s) [noise=true , address=156.146.43.225:4449 , piece_id=ZVXMQF7BR7FMGBFFFF5UPAPACIOIWNOUK3JCOG62VOG4D24FC3TQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil> Jul 26 23:45:35.421284307 (1h53m51.202217675s) [noise=true , address=104.36.82.52:4107 , piece_id=DQKOQZXURQQDTVR6ELRV5CRIFXFWJFSIMHCIJSGKDXZKV5A6C7PA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: write tcp 159.223.143.39:45372->104.36.82.52:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421302414 (1h53m51.202235796s) [noise=true , address=104.36.82.55:4199 , piece_id=RDOM7TE4TMKCIJF7CWLO6L2HOO7HNTEOLEWK3YLXXJ263CEZP4GQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:45:35.421406438 (1h53m51.202339807s) [noise=true , address=104.36.82.13:4200 , piece_id=HSLZPW5YURREZAGGW5LQN7QTDIKB4WX54IDJI4MFHMV3WILAC6NQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12dfLumooVsDsWXqi5FQMDutkcexM9A5Kmu4Vo9s1TqfWxv5ozH, address:104.36.82.13:4200): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:45:35.421690535 (1h53m51.202623904s) [noise=true , address=104.36.82.47:4522 , piece_id=VMPSGRVEATIN2IKU7MBPJTWW4DWKKGOWDBWYSLER7T3J3HDMPB3A, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421736944 (1h53m51.202670307s) [noise=true , address=104.36.82.53:4149 , piece_id=5I4OY4D4WBZA3FB7HLKCPWDQAI7TKPFDP3IECR5J6TVHQ7HQ2K5Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421795354 (1h53m51.20272872s) [noise=true , address=104.36.82.56:4247 , piece_id=2FDHHAMOQO6RGUFXLQH4UKNHYPOFYZTPEABXDHST6RVWJC5CJYCA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: rpc: dial tcp 104.36.82.56:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421847011 (1h53m51.202780377s) [noise=true , address=104.36.82.46:4490 , piece_id=QFKKQU7UEDERMIXEIWGAEAUMRCGUANOC5ZMM7HOWUBFBFZL6EQIA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: rpc: dial tcp 104.36.82.46:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:35.421871433 (1h53m51.202804809s) [seg_pos={0 0}, upload=1726]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:45:35.451883786 (1h53m51.232817154s) [upload=1726]: Done uploading segments... Jul 26 23:45:35.451947357 (1h53m51.232880736s) [upload=1726]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.033817297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [.....X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032771051s FAIL (2025-07-26 23:45:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.032795171s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 30.033294463s FAIL (2025-07-26 23:45:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 0552 (1h52m54.191933919s) [upload=1728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:44:38.411008360 (1h52m54.191941737s) [upload=1728]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:44:38.447363024 (1h52m54.228296409s) [seg_pos={0 0}, upload=1728]: Begin upload segment... Jul 26 23:44:38.447442037 (1h52m54.228375417s) [seg_pos={0 0}, upload=1728]: Done begin upload segment. Jul 26 23:44:38.447472505 (1h52m54.228405885s) [noise=true , address=77.90.31.11:4553 , piece_id=ZO7U2H2GQZUVSXM52YIFFIRPLQMFCQ24N2PVLJBZU7XFLR4H37FQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.447530743 (1h52m54.228464133s) [noise=true , address=104.36.82.7:4912 , piece_id=JCJA4K6XVTUZY5UREEOTVGO4TKY5LCAWGCQWUBFY42W33CHCCLFQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.448168457 (1h52m54.229101841s) [noise=true , address=104.36.82.13:4187 , piece_id=6JBRSVVGZE335WKHZTOTI3YWORTCDJPYKSJHUMW3A2UER2YKOA5Q, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.448218182 (1h52m54.229151566s) [noise=true , address=45.91.116.7:4011 , piece_id=BSGIRDLDXT7QMTOE2AQXLL7SF5V5CD2DTXO7ZJMJBCHG7ZVBADIQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.448712998 (1h52m54.229646397s) [noise=true , address=104.36.82.12:4150 , piece_id=VXNVNZLNJTT3CGZC4MEI3NMYUVIZYN2JTW7YZRUP2LFI43CDLOAQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.448734552 (1h52m54.229668042s) [noise=true , address=45.91.116.10:4081 , piece_id=YPIRVCBM7YS3QFDUHK62GYEJJ56V4CIKJ5PCLIGZ4WNMNYTC6VOA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449124782 (1h52m54.230058176s) [noise=true , address=104.36.82.16:4294 , piece_id=OQJOQ5ZIV2GKUEOKJPRXFT7F2E37IGBLKV5AMTLD2B4QL5SPP7EQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449223824 (1h52m54.230157203s) [noise=true , address=45.91.116.8:4047 , piece_id=V3ZDAKPXAYZPCHMPF2SWKJ22EPNF5JNSQA3JZQ5NXBAVHYTKRT2A, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449621031 (1h52m54.23055442s) [noise=true , address=104.36.82.9:4005 , piece_id=GXYGGBP3AGHKXAADRPVIGWQPGHNQEJ4KKFLVRCN36GC3KWULZLVA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449737624 (1h52m54.23067101s) [noise=true , address=77.90.31.17:4770 , piece_id=NB6VD2P6PB3JIYL3DM4SCMYBVKKKLMABOCS7JOMMLTRQIAH4UXOQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.449998923 (1h52m54.230932312s) [noise=true , address=104.36.82.2:4731 , piece_id=5UTPIJXRGD6D4XA47KYE46SY432B6KJV2QQXG2Q2OBNPKRFT2EIA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.450229755 (1h52m54.231163142s) [noise=true , address=104.36.82.3:4767 , piece_id=VYKOLINBOKDE44IKVZOSJKNRE76FBNQZ3JATRB2X2E2LBEQXORKQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.450770002 (1h52m54.23170343s) [noise=true , address=104.36.82.12:4149 , piece_id=XTIJK66IAMVTYWTJS6GY54LFTDEIEVQLXMEVEENCFIFAJ4MASDNA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.451012447 (1h52m54.23194584s) [noise=true , address=104.36.82.11:4114 , piece_id=NFTK6Q7L32SGS3A2QF65B46NKNMB7C5EBHPZ2LHMN6ZKCAEOW5QA, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.451269660 (1h52m54.232203049s) [noise=true , address=77.90.31.16:4733 , piece_id=5ABNUXZUVCJVXRFNINT6PBS43EYM57LJLUGIMJ6Y5DIHYEJA2GUQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.451502954 (1h52m54.232436341s) [noise=true , address=45.91.116.12:4225 , piece_id=EYPXNTNWCFI7ZAOA5AN3FMHIBK6UDNJGBIEFKCY5NKMZVT25E2AQ, seg_pos={0 0}, upload=1728]: Uploading piece... Jul 26 23:44:38.561793425 (1h52m54.342726809s) [upload=1728]: Next returned nil segment Jul 26 23:44:38.561815173 (1h52m54.34274856s) [upload=1728]: Waiting for error group managing segments... Jul 26 23:44:38.570459813 (1h52m54.351393192s) [noise=true , address=45.91.116.10:4081 , piece_id=YPIRVCBM7YS3QFDUHK62GYEJJ56V4CIKJ5PCLIGZ4WNMNYTC6VOA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.571277426 (1h52m54.352210805s) [noise=true , address=77.90.31.17:4770 , piece_id=NB6VD2P6PB3JIYL3DM4SCMYBVKKKLMABOCS7JOMMLTRQIAH4UXOQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.571419049 (1h52m54.352352426s) [noise=true , address=45.91.116.12:4225 , piece_id=EYPXNTNWCFI7ZAOA5AN3FMHIBK6UDNJGBIEFKCY5NKMZVT25E2AQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.571956093 (1h52m54.352889472s) [noise=true , address=77.90.31.11:4553 , piece_id=ZO7U2H2GQZUVSXM52YIFFIRPLQMFCQ24N2PVLJBZU7XFLR4H37FQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.572036717 (1h52m54.352970086s) [noise=true , address=45.91.116.7:4011 , piece_id=BSGIRDLDXT7QMTOE2AQXLL7SF5V5CD2DTXO7ZJMJBCHG7ZVBADIQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:44:38.572566661 (1h52m54.353500038s) [noise=true , address=45.91.116.8:4047 , piece_id=V3ZDAKPXAYZPCHMPF2SWKJ22EPNF5JNSQA3JZQ5NXBAVHYTKRT2A, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil> Jul 26 23:45:08.379149593 (1h53m24.16008296s) [noise=true , address=104.36.82.13:4187 , piece_id=6JBRSVVGZE335WKHZTOTI3YWORTCDJPYKSJHUMW3A2UER2YKOA5Q, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379256562 (1h53m24.160189944s) [noise=true , address=104.36.82.16:4294 , piece_id=OQJOQ5ZIV2GKUEOKJPRXFT7F2E37IGBLKV5AMTLD2B4QL5SPP7EQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379323343 (1h53m24.160256722s) [noise=true , address=104.36.82.9:4005 , piece_id=GXYGGBP3AGHKXAADRPVIGWQPGHNQEJ4KKFLVRCN36GC3KWULZLVA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379393923 (1h53m24.160327286s) [noise=true , address=104.36.82.3:4767 , piece_id=VYKOLINBOKDE44IKVZOSJKNRE76FBNQZ3JATRB2X2E2LBEQXORKQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:46344->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379446117 (1h53m24.160379495s) [noise=true , address=104.36.82.7:4912 , piece_id=JCJA4K6XVTUZY5UREEOTVGO4TKY5LCAWGCQWUBFY42W33CHCCLFQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379515407 (1h53m24.160448787s) [noise=true , address=104.36.82.12:4149 , piece_id=XTIJK66IAMVTYWTJS6GY54LFTDEIEVQLXMEVEENCFIFAJ4MASDNA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:49236->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379578632 (1h53m24.160511996s) [noise=true , address=104.36.82.11:4114 , piece_id=NFTK6Q7L32SGS3A2QF65B46NKNMB7C5EBHPZ2LHMN6ZKCAEOW5QA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379653427 (1h53m24.160586806s) [noise=true , address=77.90.31.16:4733 , piece_id=5ABNUXZUVCJVXRFNINT6PBS43EYM57LJLUGIMJ6Y5DIHYEJA2GUQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379668067 (1h53m24.160601453s) [noise=true , address=104.36.82.2:4731 , piece_id=5UTPIJXRGD6D4XA47KYE46SY432B6KJV2QQXG2Q2OBNPKRFT2EIA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379736540 (1h53m24.160669921s) [noise=true , address=104.36.82.12:4150 , piece_id=VXNVNZLNJTT3CGZC4MEI3NMYUVIZYN2JTW7YZRUP2LFI43CDLOAQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:08.379804090 (1h53m24.160737483s) [seg_pos={0 0}, upload=1728]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:45:08.410118785 (1h53m24.191052163s) [upload=1728]: Done uploading segments... Jul 26 23:45:08.410149846 (1h53m24.191083224s) [upload=1728]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.033321456s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.........] duration: 2.641604534s FAIL (2025-07-26 23:45:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.641768951s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX........] duration: 4.863295244s FAIL (2025-07-26 23:44:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.863320337s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [X.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030629469s FAIL (2025-07-26 23:44:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ZHPJXGXZTZUIULDOD2DK7ISAEB5NQWW6Q7Z3IF7Q6GTLT6AYSQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.391716684 (1h51m51.172650059s) [noise=true , address=77.90.31.11:4579 , piece_id=PGGZXX3UF4YRHNGSHBPWAC3KC6XP3OP7N3HP5BLEY3A4XFBVIUHA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392142947 (1h51m51.173076325s) [noise=true , address=77.90.31.15:4722 , piece_id=2ZFNJBPL5E6RQIGHTZDAHXA665ZP3WSFJUD4WZO72WJTFZNM3N4A, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392207849 (1h51m51.173141233s) [noise=true , address=77.90.31.19:4871 , piece_id=JNLQDO3KD3ESRGHTHC3VQHASJECGGJ33IN5N5SI7KGXXUSQ6A47A, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392518074 (1h51m51.173451449s) [noise=true , address=104.36.82.55:4218 , piece_id=ON3FC4E7PWTMO65YKFBWRKHJ6DCDEUYKGUMGVDXA35SHA3ECM5WA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392555483 (1h51m51.173488869s) [noise=true , address=104.36.82.46:4504 , piece_id=MHI5OCHIDHTFKYW7WOSOOW6II2DWP6TSH3RVJ4H52UWT3SPWKWLA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.392790318 (1h51m51.173723696s) [noise=true , address=77.90.31.21:4912 , piece_id=IG4UTY33ZJWL4FAFTLO4V72KMAUMOYARYZKOTH7JWKLNCW7J43GA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.393129281 (1h51m51.174062655s) [noise=true , address=104.36.82.11:4131 , piece_id=4YUWTYDBI6JY5XHAD4TFORRWO565FIN2UCK54YOFWZMCQEMKXGIQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.395679464 (1h51m51.176612849s) [noise=true , address=79.127.219.38:4009 , piece_id=PI3QKMMURZP4UPSTJVYSVLLPLPMB7LAL4LOVI7KJVMO3JFRIPWKQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.397122308 (1h51m51.178055693s) [noise=true , address=104.36.82.56:4245 , piece_id=5WXBJXSZ2W3YSJIPEEA2X4AP5QF5ABPS2M2UEHF7W3EQICMGARDA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.398564287 (1h51m51.179497675s) [noise=true , address=64.49.13.137:4502 , piece_id=BLP7EUD6IOLINXDNIQDONHZUS5ZPQYPBGYJZMVER7QE7IJJIDDUQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.399672121 (1h51m51.180605503s) [noise=true , address=79.127.205.233:4642 , piece_id=BV3ICMTEQXZFE4NQ2O5ESRCAIV6G7BV2YQ5F4WIHA4IO47I425MQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.400452722 (1h51m51.181386098s) [noise=true , address=79.127.220.101:4041 , piece_id=562KM7QRHXSAE4O2A4I2HEZEUJQGD63J4AFEDJDQOEYKIZFLYZ2A, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.401028982 (1h51m51.181962343s) [noise=true , address=79.127.201.215:4357 , piece_id=YUCRCTBNNTUZB46AHTWVWCW32N5ANN2FWMGFLMYETZTWJXGLRIJA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.401558656 (1h51m51.182492043s) [noise=true , address=77.90.31.16:4765 , piece_id=ODOACHSC67Q3XGEBQBDGIJE5UXWFYUPQNCGR7CLJU5FGFFGA4FKA, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.402137293 (1h51m51.183070679s) [noise=true , address=104.36.82.53:4152 , piece_id=5YY4SRMO5G4MOHFEFYRT35CS3ZII35UZ2HLOUWBNFQVCFR2XYFGQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.402671349 (1h51m51.183604737s) [noise=true , address=104.36.82.47:4535 , piece_id=WYVLKBWVMCPBA2XNH6EBBZJQJ2OWGWUPDC5KNN2MVI4F3RUFDTWQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.403312128 (1h51m51.184245514s) [noise=true , address=79.127.201.214:4321 , piece_id=57H6EJTATVVHPTPVSU4ZGJM3YW6KFRBFDFCN4V3HHBXYDHSHKJHQ, seg_pos={0 0}, upload=1713]: Uploading piece... Jul 26 23:43:35.452911857 (1h51m51.233845233s) [upload=1713]: Next returned nil segment Jul 26 23:43:35.452931776 (1h51m51.233865162s) [upload=1713]: Waiting for error group managing segments... Jul 26 23:43:35.479860829 (1h51m51.260794205s) [noise=true , address=77.90.31.11:4579 , piece_id=PGGZXX3UF4YRHNGSHBPWAC3KC6XP3OP7N3HP5BLEY3A4XFBVIUHA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:35.499783881 (1h51m51.280717248s) [noise=true , address=79.127.219.38:4009 , piece_id=PI3QKMMURZP4UPSTJVYSVLLPLPMB7LAL4LOVI7KJVMO3JFRIPWKQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:35.550152358 (1h51m51.331085733s) [noise=true , address=79.127.201.214:4321 , piece_id=57H6EJTATVVHPTPVSU4ZGJM3YW6KFRBFDFCN4V3HHBXYDHSHKJHQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:43:35.969806259 (1h51m51.750739873s) [noise=true , address=79.127.205.233:4642 , piece_id=BV3ICMTEQXZFE4NQ2O5ESRCAIV6G7BV2YQ5F4WIHA4IO47I425MQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:36.019191602 (1h51m51.800124986s) [noise=true , address=79.127.205.245:4236 , piece_id=4NZHPJXGXZTZUIULDOD2DK7ISAEB5NQWW6Q7Z3IF7Q6GTLT6AYSQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:39.604338610 (1h51m55.385271992s) [noise=true , address=77.90.31.19:4871 , piece_id=JNLQDO3KD3ESRGHTHC3VQHASJECGGJ33IN5N5SI7KGXXUSQ6A47A, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:39.927525895 (1h51m55.708459263s) [noise=true , address=79.127.201.215:4357 , piece_id=YUCRCTBNNTUZB46AHTWVWCW32N5ANN2FWMGFLMYETZTWJXGLRIJA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:41.343440817 (1h51m57.124374198s) [noise=true , address=79.127.220.101:4041 , piece_id=562KM7QRHXSAE4O2A4I2HEZEUJQGD63J4AFEDJDQOEYKIZFLYZ2A, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:41.876631082 (1h51m57.657564469s) [noise=true , address=64.49.13.137:4502 , piece_id=BLP7EUD6IOLINXDNIQDONHZUS5ZPQYPBGYJZMVER7QE7IJJIDDUQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:43:53.156195743 (1h52m8.937129124s) [noise=true , address=77.90.31.21:4912 , piece_id=IG4UTY33ZJWL4FAFTLO4V72KMAUMOYARYZKOTH7JWKLNCW7J43GA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: read tcp 159.223.143.39:54224->77.90.31.21:4912: read: connection reset by peer; conn closed: op Jul 26 23:44:16.393028126 (1h52m32.173961493s) [noise=true , address=77.90.31.15:4722 , piece_id=2ZFNJBPL5E6RQIGHTZDAHXA665ZP3WSFJUD4WZO72WJTFZNM3N4A, seg_pos={0 0}, upload=1713]: Done uploading piece. err: <nil> Jul 26 23:44:35.326069443 (1h52m51.107002823s) [noise=true , address=104.36.82.11:4131 , piece_id=4YUWTYDBI6JY5XHAD4TFORRWO565FIN2UCK54YOFWZMCQEMKXGIQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: rpc: dial tcp 104.36.82.11:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326142016 (1h52m51.107075392s) [noise=true , address=104.36.82.47:4535 , piece_id=WYVLKBWVMCPBA2XNH6EBBZJQJ2OWGWUPDC5KNN2MVI4F3RUFDTWQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326212147 (1h52m51.107145524s) [noise=true , address=104.36.82.46:4504 , piece_id=MHI5OCHIDHTFKYW7WOSOOW6II2DWP6TSH3RVJ4H52UWT3SPWKWLA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: rpc: dial tcp 104.36.82.46:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326282132 (1h52m51.107215508s) [noise=true , address=77.90.31.16:4765 , piece_id=ODOACHSC67Q3XGEBQBDGIJE5UXWFYUPQNCGR7CLJU5FGFFGA4FKA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK, address:77.90.31.16:4765): protocol: read tcp 159.223.143.39:49458->77.90.31.16:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326290240 (1h52m51.107223607s) [noise=true , address=104.36.82.56:4245 , piece_id=5WXBJXSZ2W3YSJIPEEA2X4AP5QF5ABPS2M2UEHF7W3EQICMGARDA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:12YnxyH9TZVuVheQpH9NVRfFUeWHJEtV97wyEJafhCzGZSv13Fn, address:104.36.82.56:4245): protocol: rpc: dial tcp 104.36.82.56:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326298451 (1h52m51.10723182s) [noise=true , address=104.36.82.55:4218 , piece_id=ON3FC4E7PWTMO65YKFBWRKHJ6DCDEUYKGUMGVDXA35SHA3ECM5WA, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: rpc: dial tcp 104.36.82.55:4218: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326336204 (1h52m51.10726958s) [noise=true , address=104.36.82.53:4152 , piece_id=5YY4SRMO5G4MOHFEFYRT35CS3ZII35UZ2HLOUWBNFQVCFR2XYFGQ, seg_pos={0 0}, upload=1713]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: rpc: dial tcp 104.36.82.53:4152: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:35.326363682 (1h52m51.10729706s) [seg_pos={0 0}, upload=1713]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:44:35.354528253 (1h52m51.135461628s) [upload=1713]: Done uploading segments... Jul 26 23:44:35.354583407 (1h52m51.135516784s) [upload=1713]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.030654169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 48 out of 60 is failed (threshold=3) [......X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.034044668s FAIL (2025-07-26 23:44:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.034066538s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 30.036518242s FAIL (2025-07-26 23:44:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== :43:54.157765259 (1h52m9.938698635s) [upload=1718]: Done starting upload Jul 26 23:43:54.157859239 (1h52m9.938792605s) [upload=1718]: Uploading segments... Jul 26 23:43:54.158353784 (1h52m9.939287161s) [upload=1718]: Got next segment. Inline: false Jul 26 23:43:54.158396626 (1h52m9.939330002s) [upload=1718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:43:54.158403266 (1h52m9.939336643s) [upload=1718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:43:54.270125362 (1h52m10.051058751s) [seg_pos={0 0}, upload=1718]: Begin upload segment... Jul 26 23:43:54.270192378 (1h52m10.051125767s) [seg_pos={0 0}, upload=1718]: Done begin upload segment. Jul 26 23:43:54.270246643 (1h52m10.051180023s) [noise=true , address=104.36.82.6:4874 , piece_id=D3T765DB4TZ2P3M7X5NJPFTUUKJTFHSQQOPK7DQSHGXYGMAWI4UQ, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.270259443 (1h52m10.05119283s) [noise=true , address=104.36.82.2:4731 , piece_id=5MYU2ZPKYWTBBZNDRQ64VX7VUIDJ2CTVB7RKLFLEIEKIYF7TDTPA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.270717120 (1h52m10.051650504s) [noise=true , address=45.91.116.6:4973 , piece_id=OEIDGH5JAOSTKOUS3Y5SSJKRIHK4D6RZX37NFMWJVGKWRZ75GGQA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.270794603 (1h52m10.051727983s) [noise=true , address=77.90.31.16:4734 , piece_id=QHJ37CVOP3IFGQUUEZWSYFMD2ZAZLF3T7CUV4XI4RHOJUXW35CEQ, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.271154154 (1h52m10.05208754s) [noise=true , address=45.91.116.4:4937 , piece_id=H6DG5P7GOZ44G6MJNVT4UJ4ZJXPGFUBOWLRGGH3YDIAIYT44FY5Q, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.271231647 (1h52m10.052165015s) [noise=true , address=45.91.116.8:4047 , piece_id=O7ZWWN5G6AOLUJYZLM3L3QTZ4AI5VHND33XITRPIWN5T6WRSGHMA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.271543638 (1h52m10.052477024s) [noise=true , address=104.36.82.5:4839 , piece_id=JTG3HD3JALUU3C5TYG7BXNTMM56LSQ3HZQFN5NRU5EVES4VRUXSA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.271903071 (1h52m10.052836456s) [noise=true , address=77.90.31.18:4806 , piece_id=UBFJV4OQLHH65H7MRPNRGO5AK5F4YV25K2EQCVKMT6ZTM7MX46HQ, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.272355961 (1h52m10.05328934s) [noise=true , address=104.36.82.16:4293 , piece_id=N4L3ZUWX6CXAEKBJQHYAFALP7QM7PYM467PFU7Y2YAC6WENBIVMA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.272735948 (1h52m10.053669333s) [noise=true , address=104.36.82.12:4151 , piece_id=EXYORV37DH4P7FCRQILTM3Q2FVG2BTHZXN3RM2B6PEQHUISZ6KPQ, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.272801587 (1h52m10.053734965s) [noise=true , address=45.91.116.9:4263 , piece_id=6UBRIDEXLIKU43HCQFCHFZH34HOGLYUE76UGSVCDFWYKLYRSH4BA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.273177555 (1h52m10.054110932s) [noise=true , address=104.36.82.13:4187 , piece_id=C3CXAWPGYAZ6RCTJXIQLJSNMRMSH2XGHR2RSPL3OQTT6AMHQKFNA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.273500189 (1h52m10.054433553s) [noise=true , address=77.90.31.14:4663 , piece_id=TP7YWEMBTCDJEXO3U54IK2645BVHGGLQYKCIUVVON6YLKV3UHX7A, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.273775215 (1h52m10.05470859s) [noise=true , address=104.36.82.5:4838 , piece_id=VLMXD4IX5XLQZ2627JRMHBKEQA5YWPQC3BDMCH277SMHCXWNHJWA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.274072283 (1h52m10.055005701s) [noise=true , address=45.91.116.13:4155 , piece_id=66FX6BAFHG7IYZMIBLR4W7PDDW5YGTJQUOUF7RLYWONYGNUWTVHA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.274141720 (1h52m10.05507511s) [noise=true , address=104.36.82.4:4802 , piece_id=VTTI6JL6AMHJBSUEXGAYSIPUTJQELYQYYNSORDH6QIP4AKAY4HBA, seg_pos={0 0}, upload=1718]: Uploading piece... Jul 26 23:43:54.345162001 (1h52m10.126095391s) [upload=1718]: Next returned nil segment Jul 26 23:43:54.345182867 (1h52m10.126116245s) [upload=1718]: Waiting for error group managing segments... Jul 26 23:43:54.349720399 (1h52m10.130653764s) [noise=true , address=45.91.116.13:4155 , piece_id=66FX6BAFHG7IYZMIBLR4W7PDDW5YGTJQUOUF7RLYWONYGNUWTVHA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.349743570 (1h52m10.130676952s) [noise=true , address=45.91.116.6:4973 , piece_id=OEIDGH5JAOSTKOUS3Y5SSJKRIHK4D6RZX37NFMWJVGKWRZ75GGQA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.350301115 (1h52m10.13123448s) [noise=true , address=45.91.116.8:4047 , piece_id=O7ZWWN5G6AOLUJYZLM3L3QTZ4AI5VHND33XITRPIWN5T6WRSGHMA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.350324283 (1h52m10.131257652s) [noise=true , address=45.91.116.4:4937 , piece_id=H6DG5P7GOZ44G6MJNVT4UJ4ZJXPGFUBOWLRGGH3YDIAIYT44FY5Q, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.350735883 (1h52m10.131669249s) [noise=true , address=77.90.31.14:4663 , piece_id=TP7YWEMBTCDJEXO3U54IK2645BVHGGLQYKCIUVVON6YLKV3UHX7A, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.350749964 (1h52m10.131683325s) [noise=true , address=77.90.31.18:4806 , piece_id=UBFJV4OQLHH65H7MRPNRGO5AK5F4YV25K2EQCVKMT6ZTM7MX46HQ, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:43:54.462953503 (1h52m10.243886864s) [noise=true , address=45.91.116.9:4263 , piece_id=6UBRIDEXLIKU43HCQFCHFZH34HOGLYUE76UGSVCDFWYKLYRSH4BA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: <nil> Jul 26 23:44:24.119156911 (1h52m39.900090292s) [noise=true , address=77.90.31.16:4734 , piece_id=QHJ37CVOP3IFGQUUEZWSYFMD2ZAZLF3T7CUV4XI4RHOJUXW35CEQ, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119243688 (1h52m39.900177076s) [noise=true , address=104.36.82.13:4187 , piece_id=C3CXAWPGYAZ6RCTJXIQLJSNMRMSH2XGHR2RSPL3OQTT6AMHQKFNA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119314962 (1h52m39.900248342s) [noise=true , address=104.36.82.12:4151 , piece_id=EXYORV37DH4P7FCRQILTM3Q2FVG2BTHZXN3RM2B6PEQHUISZ6KPQ, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119387866 (1h52m39.900321246s) [noise=true , address=104.36.82.5:4839 , piece_id=JTG3HD3JALUU3C5TYG7BXNTMM56LSQ3HZQFN5NRU5EVES4VRUXSA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119400276 (1h52m39.90033365s) [noise=true , address=104.36.82.2:4731 , piece_id=5MYU2ZPKYWTBBZNDRQ64VX7VUIDJ2CTVB7RKLFLEIEKIYF7TDTPA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119573282 (1h52m39.900506676s) [noise=true , address=104.36.82.6:4874 , piece_id=D3T765DB4TZ2P3M7X5NJPFTUUKJTFHSQQOPK7DQSHGXYGMAWI4UQ, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119645065 (1h52m39.900578446s) [noise=true , address=104.36.82.5:4838 , piece_id=VLMXD4IX5XLQZ2627JRMHBKEQA5YWPQC3BDMCH277SMHCXWNHJWA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119722126 (1h52m39.900655508s) [noise=true , address=104.36.82.4:4802 , piece_id=VTTI6JL6AMHJBSUEXGAYSIPUTJQELYQYYNSORDH6QIP4AKAY4HBA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:52390->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119755960 (1h52m39.900689343s) [noise=true , address=104.36.82.16:4293 , piece_id=N4L3ZUWX6CXAEKBJQHYAFALP7QM7PYM467PFU7Y2YAC6WENBIVMA, seg_pos={0 0}, upload=1718]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:44:24.119816196 (1h52m39.900749575s) [seg_pos={0 0}, upload=1718]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:44:24.152763548 (1h52m39.933696917s) [upload=1718]: Done uploading segments... Jul 26 23:44:24.152787415 (1h52m39.933720792s) [upload=1718]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.03653901s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...XX...X...X..XX.X...X..X.......X...X....XX..............X.] duration: 29.181859737s FAIL (2025-07-26 23:44:24 UTC) context deadline exceeded Duration: 29.181876506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.......] duration: 2.828121224s FAIL (2025-07-26 23:44:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.828146534s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [.......X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031747019s FAIL (2025-07-26 23:44:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031779289s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [....XX...X...X..XX.X...X..X.......X...X....XX..............X] duration: 1m0.001520363s FAIL (2025-07-26 23:43:55 UTC) context deadline exceeded Duration: 1m0.001546332s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 30.027633709s FAIL (2025-07-26 23:43:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 4s) [upload=1710]: Starting upload Jul 26 23:43:24.053285638 (1h51m39.834219013s) [upload=1710]: Done starting upload Jul 26 23:43:24.053401065 (1h51m39.834334428s) [upload=1710]: Uploading segments... Jul 26 23:43:24.053530822 (1h51m39.83446419s) [upload=1710]: Got next segment. Inline: false Jul 26 23:43:24.053574567 (1h51m39.834507942s) [upload=1710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:43:24.053580169 (1h51m39.834513544s) [upload=1710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:43:24.085282189 (1h51m39.866215564s) [seg_pos={0 0}, upload=1710]: Begin upload segment... Jul 26 23:43:24.085328954 (1h51m39.86626232s) [seg_pos={0 0}, upload=1710]: Done begin upload segment. Jul 26 23:43:24.085367234 (1h51m39.866300615s) [noise=true , address=104.36.82.3:4765 , piece_id=SJXRFQ3BCN4SSNERWLVXKR7D2OHIILNRIL3YYFINTZQHMU5K2FHQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.086026246 (1h51m39.866959633s) [noise=true , address=104.36.82.13:4187 , piece_id=MGSDRD2W4BPIQN2LFRMOWRNSOFMHBD46BFIWILZD6OE4FZHFL75Q, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.086080520 (1h51m39.867013904s) [noise=true , address=45.91.116.10:4082 , piece_id=6Q2ZG5QWFBWA7MZEXVW7CAGYAKZ2JHXD6ZGCAMPRA3PPEDYOPE4A, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.087104314 (1h51m39.868037698s) [noise=true , address=104.36.82.12:4151 , piece_id=O7VUC3F54EUMNWSQJW62UCKFOJUTBTQGSJ7JEMI3VH5IRBNZ7DJA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.087162462 (1h51m39.868095847s) [noise=true , address=45.91.116.7:4010 , piece_id=XMJL46CSVMHHNLFKSMSTAVFSKMO5XIEGTUEBGT47HPEO7NLOBAIA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.088345958 (1h51m39.869279337s) [noise=true , address=104.36.82.8:4970 , piece_id=M6T2NAYIB2DW7WI5W6INDUJ7WYJRZ4OWCVQKX6ZTMADOGYVX7ZEQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.088586179 (1h51m39.869519561s) [noise=true , address=104.36.82.7:4911 , piece_id=ZZYBMRSP3EDB656PLBHPO5XVIZBEVBJTGCC66CRTOHH5WXX5O52Q, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.089178665 (1h51m39.8701121s) [noise=true , address=45.91.116.5:4297 , piece_id=WPPROWUKTRZIOFX77EFGHTBVIMHQVTWURA5ZURFXDSNEIXRZ47RA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.089233624 (1h51m39.870167007s) [noise=true , address=45.91.116.9:4261 , piece_id=SGHKYPAS5DRZXTXNN2WV3BEUKR5V23YNF4CU23BZBXDMJIIMP5SQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.089707629 (1h51m39.870641017s) [noise=true , address=104.36.82.6:4875 , piece_id=XDS7O46ZYGEW4ICUKHUBU4JPLKE5LBJ5ZHKDH74R6TDKPG3VXMLQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.089760611 (1h51m39.87069399s) [noise=true , address=104.36.82.2:4730 , piece_id=DFAMBBZWDG66X437Z3C4IFSVB7RA5VSOMJI266WPFEQDEFPJOGTQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.090148676 (1h51m39.871082059s) [noise=true , address=104.36.82.12:4150 , piece_id=KNDZDHDXLDTZKULKDPUTPU3XERXIMIQIWI374ZJGMTG7KWJZDGDA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.090338084 (1h51m39.871271464s) [noise=true , address=77.90.31.14:4661 , piece_id=4V4SKGXUKFW5OCXNXO2HGIXJERAQRLHLFEWGV47JEV6DLNBJ6AOQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.090583734 (1h51m39.871517111s) [noise=true , address=104.36.82.10:4079 , piece_id=ZVQ2CEJWNEBJ7ZWNCK4C7PRJ4L7SATME3L2ZW3NEYFM3SNCGTDFQ, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.090979454 (1h51m39.871912835s) [noise=true , address=45.91.116.12:4226 , piece_id=LWEDERCKIHG7A4CKMILFZPEY7BDP3UXHASHAZHQUHGKPHM3MCQRA, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.091435268 (1h51m39.872368653s) [noise=true , address=45.91.116.13:4155 , piece_id=HJUWMD4G5CGV2I6JUTGBOFQRYC3PBC3YHMC4WYKL662HZA74I55Q, seg_pos={0 0}, upload=1710]: Uploading piece... Jul 26 23:43:24.190964403 (1h51m39.97189778s) [upload=1710]: Next returned nil segment Jul 26 23:43:24.190982901 (1h51m39.97191628s) [upload=1710]: Waiting for error group managing segments... Jul 26 23:43:24.194466242 (1h51m39.975399625s) [noise=true , address=45.91.116.7:4010 , piece_id=XMJL46CSVMHHNLFKSMSTAVFSKMO5XIEGTUEBGT47HPEO7NLOBAIA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.194667599 (1h51m39.97560096s) [noise=true , address=45.91.116.12:4226 , piece_id=LWEDERCKIHG7A4CKMILFZPEY7BDP3UXHASHAZHQUHGKPHM3MCQRA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.194861046 (1h51m39.975794428s) [noise=true , address=45.91.116.10:4082 , piece_id=6Q2ZG5QWFBWA7MZEXVW7CAGYAKZ2JHXD6ZGCAMPRA3PPEDYOPE4A, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.195604317 (1h51m39.976537696s) [noise=true , address=77.90.31.14:4661 , piece_id=4V4SKGXUKFW5OCXNXO2HGIXJERAQRLHLFEWGV47JEV6DLNBJ6AOQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.196035791 (1h51m39.976969167s) [noise=true , address=45.91.116.13:4155 , piece_id=HJUWMD4G5CGV2I6JUTGBOFQRYC3PBC3YHMC4WYKL662HZA74I55Q, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.271337016 (1h51m40.052270393s) [noise=true , address=45.91.116.9:4261 , piece_id=SGHKYPAS5DRZXTXNN2WV3BEUKR5V23YNF4CU23BZBXDMJIIMP5SQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:24.608416573 (1h51m40.389349942s) [noise=true , address=45.91.116.5:4297 , piece_id=WPPROWUKTRZIOFX77EFGHTBVIMHQVTWURA5ZURFXDSNEIXRZ47RA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: <nil> Jul 26 23:43:54.026196177 (1h52m9.807129564s) [noise=true , address=104.36.82.10:4079 , piece_id=ZVQ2CEJWNEBJ7ZWNCK4C7PRJ4L7SATME3L2ZW3NEYFM3SNCGTDFQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026329659 (1h52m9.807263035s) [noise=true , address=104.36.82.13:4187 , piece_id=MGSDRD2W4BPIQN2LFRMOWRNSOFMHBD46BFIWILZD6OE4FZHFL75Q, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026376716 (1h52m9.807310081s) [noise=true , address=104.36.82.2:4730 , piece_id=DFAMBBZWDG66X437Z3C4IFSVB7RA5VSOMJI266WPFEQDEFPJOGTQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026462281 (1h52m9.807395664s) [noise=true , address=104.36.82.12:4150 , piece_id=KNDZDHDXLDTZKULKDPUTPU3XERXIMIQIWI374ZJGMTG7KWJZDGDA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026564015 (1h52m9.807497395s) [noise=true , address=104.36.82.6:4875 , piece_id=XDS7O46ZYGEW4ICUKHUBU4JPLKE5LBJ5ZHKDH74R6TDKPG3VXMLQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026646507 (1h52m9.807579888s) [noise=true , address=104.36.82.8:4970 , piece_id=M6T2NAYIB2DW7WI5W6INDUJ7WYJRZ4OWCVQKX6ZTMADOGYVX7ZEQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026754706 (1h52m9.807688071s) [noise=true , address=104.36.82.3:4765 , piece_id=SJXRFQ3BCN4SSNERWLVXKR7D2OHIILNRIL3YYFINTZQHMU5K2FHQ, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026809577 (1h52m9.807742944s) [noise=true , address=104.36.82.12:4151 , piece_id=O7VUC3F54EUMNWSQJW62UCKFOJUTBTQGSJ7JEMI3VH5IRBNZ7DJA, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026876891 (1h52m9.807810258s) [noise=true , address=104.36.82.7:4911 , piece_id=ZZYBMRSP3EDB656PLBHPO5XVIZBEVBJTGCC66CRTOHH5WXX5O52Q, seg_pos={0 0}, upload=1710]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:54.026904979 (1h52m9.807838356s) [seg_pos={0 0}, upload=1710]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:43:54.051099432 (1h52m9.832032809s) [upload=1710]: Done uploading segments... Jul 26 23:43:54.051131056 (1h52m9.832064436s) [upload=1710]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.027654647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [XX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031471146s FAIL (2025-07-26 23:43:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 95]: Begin upload segment... Jul 26 23:42:35.282602134 (1h50m51.063535515s) [seg_pos={0 0}, upload=1695]: Done begin upload segment. Jul 26 23:42:35.282651075 (1h50m51.063584462s) [noise=true , address=77.90.31.18:4812 , piece_id=KYDR5NBFVGYT37SDVW7LIPYDQETTU63YGTBQJGRX25TOYQHDUIFA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.282675926 (1h50m51.06360931s) [noise=true , address=77.90.31.10:4543 , piece_id=DYX2L553ZJKNQGPCW2WLUEMDVHUMCEPASQ4TWW3WJ4ZFWUH7PQNA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.283305808 (1h50m51.064239193s) [noise=true , address=79.127.205.245:4223 , piece_id=OWLLV7ALZPFK43ARLQOHS5IODPHYOUVK5KJOI722JYYXK7GF3QHA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.283868138 (1h50m51.064801524s) [noise=true , address=104.36.82.52:4107 , piece_id=HAHRLH3QEW4A3ABCPWVUDJTGNLDGBJW7DB3N5V3AGUCQLMXKF3BQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.283943769 (1h50m51.064877152s) [noise=true , address=79.127.201.214:4321 , piece_id=2XR23HDME4UFYHOEPTM5ZC3BBECEOLLVCOXXKGNCJQQF7X3BPFGA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.284422849 (1h50m51.065356233s) [noise=true , address=79.127.220.101:4041 , piece_id=ZHPWOXFKTP4IKTJCWLBZSGB54PWFCA4W6SYMXUKRROF2D6WZKVEA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.284467062 (1h50m51.065400443s) [noise=true , address=77.90.31.15:4717 , piece_id=T54LAZH4S4N5KZUW73N2MTWWYJDHGHAXA2ABYPBZWBIP3NHXQAOA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.284790285 (1h50m51.065723665s) [noise=true , address=156.146.43.225:4448 , piece_id=RIPBINT4MALR7FC3RDI2BFUWYK4RZISJNB6HYCJN5X4MST3PCDLQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.284946847 (1h50m51.06588023s) [noise=true , address=104.36.82.55:4200 , piece_id=LN5DOA2DAM5GIA5JNMZCCVUUOKKMXKYGRWARBAVLZYCAGREWN6PA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.285098024 (1h50m51.066031399s) [noise=true , address=104.36.82.54:4190 , piece_id=WNW7ODGCUGS4VAISGRORFKBJCVGPIOFUSIODYE2MRDKXC6XTOZLA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.285385085 (1h50m51.06631846s) [noise=true , address=79.127.205.235:4673 , piece_id=32JJW5WDGDT57IIFQQNJE722QQD7BI2YRNTBSU6KTCTLJF26QGTQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.285463416 (1h50m51.066396801s) [noise=true , address=104.36.82.47:4523 , piece_id=TA7226R3LANDX5RMISRCJYJ646C25AFRFCEQ6ZWKTP3E75BMQGFA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.285939070 (1h50m51.066872459s) [noise=true , address=64.49.13.138:4482 , piece_id=AWXDQ3TRVC3CNNXLZAE5XJBIYYF5UUCVGVK3PKELVEKZ5TLEJVXA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.286377672 (1h50m51.067311055s) [noise=true , address=79.127.219.38:4004 , piece_id=LVSY6XTTEPS6JUPD7YTMDCKW4DZVICXXRO4OGTELHLVSKVI3GXPQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.286802945 (1h50m51.067736327s) [noise=true , address=79.127.201.215:4344 , piece_id=S326D47EJWUNM4TWSEBQ6VXCIVDFLWKMYX46AVPHD6RI4G5BPROQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.287257627 (1h50m51.068191007s) [noise=true , address=79.127.205.234:4610 , piece_id=FHUTQB5EJRF6V6TWIWGVQHFJEQBJWSWOGP7A7OMHBYJZGQPIYFTA, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.287665206 (1h50m51.068598593s) [noise=true , address=104.36.82.46:4510 , piece_id=E76TEGR5N5YOK3J4U34V4PFCQ37XRHENDUBGWRM2SXPBNUL3PRAQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.298538049 (1h50m51.079471423s) [noise=true , address=79.127.205.233:4624 , piece_id=ZBXGBEELHHHUFTFEOMNG7ZNJ7CUVJBZPEDVACVMP6QJUTH7YHQZQ, seg_pos={0 0}, upload=1695]: Uploading piece... Jul 26 23:42:35.352528873 (1h50m51.133462262s) [upload=1695]: Next returned nil segment Jul 26 23:42:35.352550245 (1h50m51.133483628s) [upload=1695]: Waiting for error group managing segments... Jul 26 23:42:35.445801385 (1h50m51.226734764s) [noise=true , address=79.127.201.214:4321 , piece_id=2XR23HDME4UFYHOEPTM5ZC3BBECEOLLVCOXXKGNCJQQF7X3BPFGA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:35.455970865 (1h50m51.236904227s) [noise=true , address=77.90.31.18:4812 , piece_id=KYDR5NBFVGYT37SDVW7LIPYDQETTU63YGTBQJGRX25TOYQHDUIFA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.705136251 (1h50m51.486069628s) [noise=true , address=79.127.219.38:4004 , piece_id=LVSY6XTTEPS6JUPD7YTMDCKW4DZVICXXRO4OGTELHLVSKVI3GXPQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.827472444 (1h50m51.608405827s) [noise=true , address=64.49.13.138:4482 , piece_id=AWXDQ3TRVC3CNNXLZAE5XJBIYYF5UUCVGVK3PKELVEKZ5TLEJVXA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:35.862584489 (1h50m51.643517866s) [noise=true , address=79.127.205.234:4610 , piece_id=FHUTQB5EJRF6V6TWIWGVQHFJEQBJWSWOGP7A7OMHBYJZGQPIYFTA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.880615994 (1h50m51.661549361s) [noise=true , address=79.127.205.245:4223 , piece_id=OWLLV7ALZPFK43ARLQOHS5IODPHYOUVK5KJOI722JYYXK7GF3QHA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.895091485 (1h50m51.676024865s) [noise=true , address=79.127.205.233:4624 , piece_id=ZBXGBEELHHHUFTFEOMNG7ZNJ7CUVJBZPEDVACVMP6QJUTH7YHQZQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:35.933695124 (1h50m51.71462849s) [noise=true , address=79.127.205.235:4673 , piece_id=32JJW5WDGDT57IIFQQNJE722QQD7BI2YRNTBSU6KTCTLJF26QGTQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:37.280783511 (1h50m53.06171688s) [noise=true , address=79.127.220.101:4041 , piece_id=ZHPWOXFKTP4IKTJCWLBZSGB54PWFCA4W6SYMXUKRROF2D6WZKVEA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:38.141031537 (1h50m53.921964913s) [noise=true , address=156.146.43.225:4448 , piece_id=RIPBINT4MALR7FC3RDI2BFUWYK4RZISJNB6HYCJN5X4MST3PCDLQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:58.856493400 (1h51m14.637426768s) [noise=true , address=79.127.201.215:4344 , piece_id=S326D47EJWUNM4TWSEBQ6VXCIVDFLWKMYX46AVPHD6RI4G5BPROQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:42:59.354096556 (1h51m15.135029936s) [noise=true , address=77.90.31.15:4717 , piece_id=T54LAZH4S4N5KZUW73N2MTWWYJDHGHAXA2ABYPBZWBIP3NHXQAOA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: <nil> Jul 26 23:43:35.221090919 (1h51m51.002024296s) [noise=true , address=104.36.82.52:4107 , piece_id=HAHRLH3QEW4A3ABCPWVUDJTGNLDGBJW7DB3N5V3AGUCQLMXKF3BQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: rpc: dial tcp 104.36.82.52:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221147081 (1h51m51.002080457s) [noise=true , address=77.90.31.10:4543 , piece_id=DYX2L553ZJKNQGPCW2WLUEMDVHUMCEPASQ4TWW3WJ4ZFWUH7PQNA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 159.223.143.39:56664->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221183240 (1h51m51.002116617s) [noise=true , address=104.36.82.47:4523 , piece_id=TA7226R3LANDX5RMISRCJYJ646C25AFRFCEQ6ZWKTP3E75BMQGFA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: rpc: dial tcp 104.36.82.47:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221229901 (1h51m51.002163277s) [noise=true , address=104.36.82.46:4510 , piece_id=E76TEGR5N5YOK3J4U34V4PFCQ37XRHENDUBGWRM2SXPBNUL3PRAQ, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: write tcp 159.223.143.39:37314->104.36.82.46:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221269138 (1h51m51.002202514s) [noise=true , address=104.36.82.55:4200 , piece_id=LN5DOA2DAM5GIA5JNMZCCVUUOKKMXKYGRWARBAVLZYCAGREWN6PA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221299563 (1h51m51.002232924s) [noise=true , address=104.36.82.54:4190 , piece_id=WNW7ODGCUGS4VAISGRORFKBJCVGPIOFUSIODYE2MRDKXC6XTOZLA, seg_pos={0 0}, upload=1695]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:35.221316944 (1h51m51.002250328s) [seg_pos={0 0}, upload=1695]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:43:35.250332304 (1h51m51.031265679s) [upload=1695]: Done uploading segments... Jul 26 23:43:35.250361467 (1h51m51.031294841s) [upload=1695]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.03150034s core uplink us-select-1 us1-placement-20

1
...
5507
5508
5509
5510
5511
...
6004